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ABSTRACT 

A  system-of-systems  engineering  approach  to  suit  the  Australian  land  force  capability 
integration  challenge  is  developed  drawing  on  the  international  body  of  knowledge.  The 
approach  proposed  would  require  only  quite  modest  resources.  The  nature  of  the  activities  and 
the  artefacts  to  produce  are  defined,  with  a  focus  on  SoS  requirements  engineering  aspects  and 
how  these  influence  the  resulting  project-  and  SoS-level  test  and  evaluation  activities. 
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A  System-of-Systems  Engineering  Approach  for 
Australian  Land  Force  Capability  Integration 

Executive  Summary 


This  report  forms  the  final  report  of  a  Research  Agreement  between  Land  Operations 
Division  of  DSTO  and  the  Defence  Systems  Innovation  Centre  (DSIC)  entitled  Research 
Project  for  SoS  Requirements  and  Evaluation.  It  builds  on  the  outline  of  an  overarching 
Systems  of  Systems  Engineering  (SoSE)  approach  for  land  force  capability  engineering 
described  in  Deliverable  1  (Cook  and  Nowakowski,  2012)  and  the  success  factors  for 
Systems  of  Systems  (SoS)  programs  enumerated  in  Deliverable  3  (Cook,  et  ah,  2012). 

The  report  opens  with  a  description  of  the  broad  land  force  capability  challenge  and  then 
focuses  on  the  central  issues  to  be  examined  in  this  report,  namely  the  provision  of  advice 
on  how  best  to  achieve: 

1 .  Capability/SoS  project  requirements  development  and  specification;  and 

2.  Test  and  Evaluation  (T&E)  processes  for  capability/SoS  integration  to  ensure  that 
the  delivered  constituent  systems  will  be  effective  components  of  land  force 
capability. 

Specifically,  it  was  agreed  that  the  requirements  engineering  and  test  and 
evaluation  advice  sought  would  need  to  be  framed  within  the  context  of  a  defined 
land  force  capability  integration  approach  and  hence  the  body  of  the  report 
commences  by  defining  the  elements  of  capability  needed  to  undertake  Army  SoS 
integration  for  the  tactical  land  force  operating  in  a  joint  environment.  These  are: 
overarching  principles  and  approach;  organisation,  roles  and  responsibilities;  SoS 
engineering  environment;  personnel;  training;  SoS  design  and  synthesis;  and  SoS 
test  and  evaluation.  Success  factors  for  each  element  of  SoSE  capability  are 
described  next  to  inform  the  design  and  implementation  of  a  SoSE  approach  for  the 
land  force. 

SoS  capabilities  can  be  implemented  in  different  ways  informed  by  equally  different 
frameworks  of  ideas.  A  description  of  the  five  leading  schools  of  thought  on  SoSE  follows 
to  provide  an  understanding  of  the  diversity  of  approaches  that  are  gaining  traction  around 
the  world.  The  starting  school  of  thought  chosen  for  the  land  force  capability  integration 
challenge  was  the  US  Department  of  Defense  (DoD)  SoSE  approach  because  it  is  the  most 
mature  in  many  respects,  is  well  documented,  is  well-known  and  respected  in  Australia 
(indeed  it  acknowledges  Australian  input),  and  is  a  defence  methodology  and  hence  well- 
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suited  to  the  problem  at  hand.  The  US  DoD  approach,  however,  has  been  developed  to  suit 
rather  different  resource  levels,  contractual  arrangements,  and  governance  models.  The 
argument  is  then  made  that  it  is  necessary  to  tailor  the  US  approach  to  suit  Australia’s  level 
of  SoSE  capability  maturity,  cultural  differences,  and  prospective  level  of  SoSE  funding 
and  a  set  of  guiding  principles  for  the  design  of  the  Australian  approach  is  then  described 
based  on  the  literature,  the  authors’  knowledge  of  the  Australian  defence  enterprise,  and  a 
review  of  the  success  factors.  The  point  is  made  that  SoSE  is  yet  to  commence  in  earnest  in 
Australia  and,  as  such,  the  focus  in  the  first  one  or  two  iterations  will  need  to  be  on 
‘herding’  the  projects  to  achieve  SoS  outcomes  and  then  to  progressively  adopt  the  precepts 
of  capability  engineering  to  shape  the  project  requirements. 

The  design  of  an  Australian  land  force  capability  SoS  approach  is  described  in  Section  4  by 
suggesting  how  the  five  US  DoD  SoSE  lifecycle  stages  can  be  adapted.  Included  in  this 
description  are  the  outputs  of  each  of  the  phases  embodied  in  the  form  of  artefacts  that 
would  need  to  be  produced  during  each  iteration.  An  Australianised  description  of  the 
artefacts  can  be  found  in  Annex  B. 

A  key  message  arising  from  the  research  that  underpins  the  findings  of  this  report  is  that 
SoSE  is  an  enterprise-level  activity  that  relies  on  stakeholder  buy-in  and  commitment  to 
achieve  success.  Success  always  depends  on  the  stakeholder  perspective  and,  after  the  SoS 
Team,  the  most  important  perspective  to  consider  is  that  of  the  constituent  system  System 
Project  Offices  (SPOs).  The  analysis  of  the  SoSE  activity  from  this  perspective  readily 
surfaces  the  need  to  incentivise  SPOs  to  develop  their  products  and  services  in  such  a  way 
as  to  maximise  SoS  outcomes.  A  description  of  the  Land  SoSE  support  environment 
completes  the  description  of  the  proposed  Australian  SoSE  approach. 

Section  5  addresses  the  first  question  explicitly.  We  show  how  capability  requirements  for 
the  SoS  are  elicited  and  how  the  SoSE  analysis  stage  identifies  how  these  can  be  mapped 
onto  constituent  systems  in  the  form  of  high-level  service  definitions  (analogous  to  service 
contracts).  In  the  first  iteration,  the  SoS  Team  will  have  little  opportunity  in  the  short  term 
to  change  constituent  systems  requirements  or  project  scope  and  hence  the  approach  chosen 
to  interface  to  the  SPOs  is  to  establish  agreements  that  embody  the  service  definitions.  It  is 
these  service  and  interface  definitions  that  are  tracked  by  both  parties  and  form  the  basis  for 
guiding  the  evolution  of  the  SoS. 

Section  6  addressed  the  second  question  explicitly.  Firstly,  SoS  T&E  is 
fundamentally  different  in  goals,  character,  and  intent  from  system-acquisition 
T&E  that  seeks  to  inform  acceptance  and  deployment  decisions.  SoS  T&E  is 
focussed  more  on  SoS  verification  and  validation  because  it  is  deemed  impractical 
to  conduct  comprehensive  testing  of  an  evolving  SoS.  Thus  SoS  T&E  activities  focus 
on  the  impacts  that  iterations  of  constituent  systems  will  make  on  the  SoS 
performance  and  behaviour  outcomes.  As  such,  it  involves  far  greater  use  of 
modelling,  simulation  and  analysis  than  project  T&E  and  thinking  of  training, 
exercises,  and  operations  as  T&E  opportunities.  Nonetheless,  the  SoS  T&E  staff 
would  monitor  the  constituent  system  project  progress  to  evaluate  whether  the 
service  provision  clauses  of  the  agreements  will  be  achieved. 
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SoSE  metrics  are  beginning  to  develop.  These  would  become  useful  when  the  Australian 
managerial  and  technical  processes  become  established.  We  suggest  that  a  metric-based 
approach  is  inappropriate  during  the  initial  iterations  of  SoSE  because  it  would  encourage 
counterproductive  stakeholder  behaviour,  and  is  covered  in  Section  7. 

In  summary,  this  report  draws  on  the  international  body  of  knowledge  to  develop  a  SoSE 
approach  to  suit  the  Australian  land  force  capability  integration  challenge.  It  uses  the  US 
DoD  approach  for  its  outline  supplemented  by  social  and  cultural  aspects  from  European 
and  civilian  experiences.  The  approach  proposed  would  require  quite  modest  resources,  “a 
few  top  people”,  supported  by  contemporary  tools,  and  some  modest  additional  funding  for 
constituent  system  SPOs.  The  nature  of  the  activities  and  the  artefacts  to  produce  are 
defined.  Specific  attention  has  been  paid  to  SoS  requirements  engineering  aspects  and  how 
these  influence  the  resulting  project-  and  SoS-level  T&E  activities.  A  final  point  is  that 
SoSE  is  quite  different  from  project  SE  and  SoS  T&E  is  fundamentally  different  from 
project-based  T&E.  Both  disciplines  will  require  highly-capable  and  adaptable  people  that 
display  the  appropriate  mind-set  needed  for  SoS  success.  Identifying  and  nurturing  the 
development  of  these  people  is  an  important  subject  for  further  consideration. 

Ultimately,  the  success  of  SoSE  will  be  judged  in  terms  of  the  achieved  SoS  capability 
performance  versus  the  cost  of  undertaking  SoSE.  A  key  return  on  investment  for  SoSE  is 
likely  to  be  a  transfer  of  the  SoS  integration  risks  from  the  operational  level  (i.e.  the 
warfighters),  where  they  have  been  traditionally  addressed,  to  the  capability  development 
domain. 
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1.  Introduction 


1.1  Background 

In  the  Land  domain,  systems  integration  presents  a  major  challenge  to  delivering  effective  and 
survivable  war  fighting  capability  ranging  from  force  level  joint  and  combined  arms  teams, 
through  close  combat  teams,  down  to  the  individual  soldier.  This  is  driven  by  a  number  of 
factors  including:  the  inherent  complexity  of  operations;  rapidly  increasing  complexity  of  the 
soldier;  vehicles  and  other  systems;  and  the  Defence  preference  for  commercial  off-the-shelf 
(COTS)  or  military  off-the-shelf  (MOTS)  acquisition.  It  has,  therefore,  become  essential  to 
better  align  many  elements  of  Defence  and  industry  to  deliver  effectively  integrated  systems 
and  capabilities  to  the  soldier  and  commanders. 

Land  force  capability  is  generally  acquired  within  individual  projects  (both  major  and  minor) 
that  will  deliver  specific  systems.  These  systems  will  not  necessarily  be  optimised  to  integrate 
into  wider  land  forces.  Thus,  there  is  a  need  to  more  clearly  understand  what  is  required  to 
achieve  land  force  capability  integration  and  how  individual  projects  can  specify  and  evaluate 
systems  such  that  they  will  be  able  to  effectively  integrate  into  the  broader  land  force 
capability  or  system-of-sy stems  (SoS). 

1.2  Scope 

This  report  is  one  of  three  reports  that  are  the  culmination  of  a  joint  research  program  between 
the  Defence  Systems  Innovation  Centre  (DISC)  and  DSTO  on  the  challenge  of  land  force 
integration  (CAPO  2011).  The  first  (Cook  and  Nowakowski,  2012)  provides  an  initial  outline 
of  an  overarching  approach  to  address  the  problem.  The  second  (Cook  et  al.,  2012)  identifies 
and  analyses  the  lessons  learned  in  implementing  SoS.  This  report  builds  on  these  two  reports 
to  deliver  advice  on  (CAPO  2011): 

1 .  Capability/SoS  project  requirements  development  and  specification;  and 

2.  Test  and  Evaluation  (T&E)  process  for  capability/SoS  integration  to  ensure  that  the 
delivered  constituent  systems  will  be  effective  components  of  land  force  capability. 

The  interpretation  of  the  intent  and  content  of  this  deliverable  was  honed  during  the  conduct 
of  the  research  task  in  a  sequence  of  project  meetings.  It  was  agreed  that  the  requirements 
engineering  and  test  and  evaluation  advice  sought  would  need  to  be  framed  within  the 
context  of  land  force  capability  integration,  and  hence  significant  effort  was  devoted  to 
outlining  a  viable  SoS  integration  (SoSI)  approach  for  the  tactical  land  force  operating  in  a 
joint  environment.  We  used  battalion-level  and  amphibious  operations  to  frame  our  thinking. 

Furthermore,  the  scope  was  clarified  to  include  the  requirements  and  T&E  at  both  the  land 
force  level  and  the  constituent  project  level. 
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2.  The  Land  Force  SoSI  Challenge 

Recent  studies  have  indicated  that  the  biggest  systems  integration  challenge  facing  the 
Australian  Defence  Organisation  is  systems-of-systems  integration  (sometimes  referred  to  as 
cross-project  integration)  (RPDE,  2011;  Defence,  2012).  System  of  Systems  Integration  (SoSI) 
refers  to  the  integration  of  interoperable  communication,  command  and  control,  and  support 
systems  between  various  platforms  which  link,  for  example,  the  Landing  Helicopter  Dock 
with  air  support,  amphibious  watercraft,  support  ships,  and  land  forces,  in  order  to  provide 
the  overarching  amphibious  capability.  SoSI  is  a  key  capability  for  realising  the  Networked 
Force  described  in  the  2009  Defence  White  Paper:  Defending  Australia  in  the  Asia  Pacific 
Century:  Force  2030  and  in  ancillary  documents  such  as  the  NCW  Roadmap  and  the  ISR 
Roadmap. 

SoSI  is  achieved  through  the  application  of  systems  engineering  tailored  to  reflect  the  SoS 
context:  there  usually  is  no  single  owner  of  a  SoS,  and  independent,  concurrent  management 
and  funding  at  both  the  constituent  system  level  and  at  the  SoS  level  is  the  norm  (Maier,  1998; 
Chen  and  Clothier,  2003;  Valerdi  et  al  2008,  OSD  2008).  Hence,  to  achieve  effective  SoSI,  it 
becomes  necessary  for  key  players  to  influence  other  parties  (in  particular  independent  SPOs) 
on  the  basis  of  perspective,  breadth  of  knowledge,  and  analysis,  rather  than  from  a  position  of 
authority  (Oxenham  and  Swales,  2010).  The  discipline  that  tackles  SoSI  is  internationally 
known  as  SoS  Engineering  (SoSE)  but  it  is  still  in  its  infancy  and  SoSE  remains  a  challenge 
throughout  the  world. 

This  report  is  informed  by  a  substantial  literature  review,  the  work  of  The  Technical  Co¬ 
operation  Program  panel  TTCP-JSA-TP4:  Systems  Engineering  for  Defence  Modernisation, 
and  several  recent  studies  in  the  defence  SoS  field.  Of  particular  relevance  to  the  design  and 
evolution  of  a  Land  SoSE  capability  is  the  description  of  SoSE  Success  Factors  that  can  be 
found  in  Cook  et  al.  (2012)  that  builds  on  a  substantial  literature  base  and  recent  Australian 
studies  (RPDE,  2011;  Defence,  2012).  Interested  readers  are  referred  to  Annex  A  for 
comprehensive  definitions  of  the  terms  such  as  SE,  SoS,  SoSI  and  SoSE. 
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3.  A  Capability-based  Approach  to  SoS  Engineering 


The  ability  to  undertake  SoSE  can  usefully  be  thought  of  as  a  capability  that,  like  a  military 
capability,  requires  the  synthesis  of  many  components  to  realise  the  desired  effect;  indeed 
Sage  and  Cuppan  (2001),  among  others,  assert  that  there  is  value  in  thinking  of  the  social 
system  that  undertakes  SoSE  as  a  complex  adaptive  system  and  using  this  stance  to  inform  its 
definition  and  evolution.  There  are  many  variations  of  the  definitions  of  the  elements  of 
military  capability:  Fundamental  Inputs  to  Capability  (Defence  2011),  PRICIE  (Hendry  2007), 
Defence  Lines  of  Development  (MOD  2009),  and  DOTMLPF  (Defence  2009a).  These  have  been 
synthesised  as  shown  in  Figure  1  to  arrive  at  a  list  of  capability  elements  for  SoSE  capability. 
These  elements  need  not  be  thought  of  as  strictly  orthogonal,  rather  they  can  be  considered  to 
be  perspectival  views  of  SoSE  enterprise.  The  figure  also  draws  on  ANSI /  EIA-632  to  indicate 
that  SoSE,  just  like  project  engineering,  operates  within  an  environment  that  sets  the  external 
and  enterprise  context  for  SoSE  planning  and  conduct  of  the  activities.  The  external 
environment  covers  laws  and  regulations,  legal  liabilities,  social  responsibility,  technology 
base,  labour  pool,  competing  products,  standards  and  specifications,  and  public  culture.  For 
the  case  in  hand,  the  enterprise  environment  covers  Defence  policies  and  procedures, 
standards  and  specifications,  guidelines,  domain  (Land  and  military  information  system 
technologies,  and  local  culture.  The  conduct  SoSE  will  also  be  shaped  by  environmental 
factors  that  will  apply  specifically  to  the  SoS,  for  example  directives  and  procedures,  plans, 
tools,  reviews  and  metrics. 
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Figure  1.  The  derivation  of  the  elements  of  SoSE  capability 
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A  brief  discussion  on  each  of  these  elements  of  SoSE  capability  follows  along  with  some  of  the 
constraints  in  which  they  need  to  operate  in  the  Australian  capability  development 
environment.  Also  included  in  each  element  is  a  list  of  success  factors  of  relevance  to  the  land 
force  capability  SoS  we  are  considering. 

3.1  Australian  SoSE  Environment 


Figure  2  illustrates  the  key  environmental  factors  that  influence  the  selection  and 
implementation  of  any  Australian  Defence  SoSE  approach.  The  Australian  Defence  Enterprise 
Environment  includes  the  capability  of  the  Australian  Defence  Systems  Engineering  and 
Integration  Enterprise  (this  includes  all  the  contributors  from  government,  contractors,  and 
academia)  as  well  as  the  technology  base,  public  culture,  the  legal  and  contractual  framework 
and  norms,  government  policies  such  as  the  Defence  Industry  Policy,  and  the  nature  and 
volume  of  the  work  that  is  in  progress  and  programmed  into  the  Defence  Capability  Plan. 


We  also  know  that  the  decision  on  the  SoSE  approach  to  use  would  need  to  be  informed  by 
government  and  defence  directives  and  policies,  coalition  and  national  operational  needs,  the 
current  and  future  Land  Defence  Outputs,  and  the  international  literature  on  SoSE,  in 
particular  the  guidebooks  and  practices. 


The  finding  of  Unewisse  and  Cook  (2011a  &  b)  coupled  with  more  investigations  (Defence, 
2012)  indicate  that  the  Australian  Defence  Enterprise  has  modest  capability  in  SoSE  and,  as 
such,  it  is  fair  to  say  that  any  approach  initiated  in  this  environment  could  expect  to  exhibit  a 
low  capability  maturity  level  and  would  rely  on  "heroes"  and  "wise  heads"  rather  than 
process  to  achieve  its  objectives  (Sage  and  Cuppan,  2001). 


Figure  2.  A  context  diagram  for  the  Australian  Land  SoSE  approach 
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3.2  Discussion  on  Overarching  Principles  and  Approach 

The  Overarching  Principles  and  Approach  includes  the  SoS  design,  management  and 
coordination  approach,  the  definition  of  requirements  for  the  other  elements,  particularly  the 
engineering  and  validation  environments,  and  key  business  processes  and  procedures.  The 
latter  define  what  SoSE  will  comprise  and  how  it  will  be  planned,  applied,  monitored,  and 
evaluated. 

Of  the  guidebooks,  the  most  referenced  body  of  knowledge  with  respect  to  military  SoS 
integration  is  the  US  Department  of  Defense's  Systems  Engineering  Guide  for  Systems  of 
Systems  (OSD,  2008)  that  is  based  on  the  study  of  18  SoSs.  It  provides  the  following  insights: 

•  SoSE  management  is  essentially  different  from,  and  more  complex  than,  single-project 
management  because  there  is  limited  authority  to  influence  the  SPOs  providing  the 
constituent  systems. 

•  SoSE  will  never  be  a  "one-size-fits-all"  process  but  rather  something  that  must  be 
developed  and  tailored  in  cognisance  of  key  SoS  constituent  systems  and  their 
lifecycles. 

•  SoSE  requires  an  emphasis  on  end-to-end  performance  and  behaviour  management  to 
ensure  that  the  ensemble  of  systems  will  meet  capability  targets. 

The  guide  identifies  Principal  SoSE  Elements,  SoSE  Principles,  and  Focus  Areas  for  SoSE 
Management  and  these,  coupled  with  later  work  on  the  "Wave  Model"  of  SoSE  (Dahmann  et 
al,  2011),  form  a  useful  starting  place  for  the  identification  of  an  overall  Australian  Land  SoSE 
approach. 

This  element  will  also  surface  and  promulgate  the  principles  that  will  inform  SoSE  practice. 
Some  example  principles  are  listed  below  that  are  adapted  from  the  US  SoSE  Guide  (OSD, 
2008): 

•  SoSE  can  only  commence  when  a  SoS  has  been  acknowledged,  roles  and 
responsibilities  have  been  defined,  and  resources  have  been  allocated  to  the  task. 

•  It  is  essential  to  focus  on  the  design  strategy  and  trades  both  when  the  formal  SoS  is 
first  established  and  throughout  the  SoS  evolution. 

•  Organizational  as  well  as  technical  issues  need  to  be  addressed  in  making  SE  trades 
and  decisions. 

•  The  different  roles  of  systems  engineers  at  the  system,  versus  the  SoS,  level  and  the 
relationship  between  the  SE  done  at  the  two  levels,  must  be  acknowledged. 

•  Success  depends  on  the  ability  of  SoS  managers  to  work  across  systems  and  balance 
technical  and  non-technical  issues.  This  requires  experienced,  capable  SoS  managers 
and  SE  teams. 

•  Use  a  SoS  solution  based  on  open  systems  techniques  and  loose  coupling. 

•  Knowledge  management  is  a  vital  enabler  for  effective  SoSE. 
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Just  as  the  definitions  of  the  elements  of  capability  differ  in  Australia  from  allied  countries  so 
will  the  definition  of  the  principles  that  will  need  to  inform  our  SoSE  practice.  Thus  the  above 
should  be  seen  as  an  example  from  which  we  can  evolve  the  final  set. 

It  should  be  recognised  that  SoSE  usually  commences  after  the  constituent  systems  are  in 
service,  or  at  least  in  development,  and  frequently  after  certain  capabilities  of  the  SoS  exist  in 
practice  (Krygiel,  1999;  Sage  and  Culpan,  2003;  Kalawsky  et  al,  2012b).  Thus  the  overarching 
SoSE  approach  should  not  be  thought  of  as  an  ab  initio  methodology  but  rather  one  intended 
to  accelerate  and  better  coordinate  the  realisation  of  the  SoS  capability  (OSD,  2008).  However, 
lessons  and  insights  from  the  initial  synthesis  efforts  can  be  used  to  shape  later  iterations  of 
the  SoS  to  support  a  migration  to  a  more  systemic  SoSE  approach. 

Note  that  SoS  exist  and  will  evolve  without  overt  SoSE,  funding,  dedicated  roles,  or  even 
acknowledgement.  These  SoS  may  still  be  shaped,  usually  based  on  the  drive  of  an  individual 
rather  than  via  the  systemic  approach  offered  by  SoSE.  The  success  of  this  approach  is 
critically  dependent  upon  the  energy  and  presence  of  the  champion  and  will  often  falter  on 
their  departure.  Also  this  approach  increasingly  struggles  to  deal  with  the  complexity  of 
modern  warfighting  capabilities  and  fails  to  leverage  the  opportunity  to  shape  the  SoS  during 
the  capability  development  and  acquisition  phases. 

There  are  many  current  SoS/ capabilities  efforts  (including  the  Land  Battlefield  Operating 
Systems  such  as,  land  manoeuvre,  land  ISR,  logistics,  engineering, . . .),  which  are  in  the  main 
only  weakly  coupled  to  the  current  acquisition  process.  The  SoS  input  for  these  capabilities 
tends  to  be  at  the  start  (concepts  and  high-level  needs)  and  end  (force  generation  and 
operations)  of  the  capability  process.  Thus,  any  implementation  of  a  SoSE  approach  needs  to 
address  how  the  current  efforts  to  identify  and  define  SoS/ capability  concepts  can  enhance 
the  SE  of  the  component  projects  to  deliver  a  more  effective  SoS.  Similarly,  the  SoSE  approach 
needs  to  provide  lessons  learnt  mechanisms  (feedback  paths)  from  the  force  generation 
process  to  assist  in  shaping  capability  requirements  and  acquisition. 

Cook  et  al  (2012),  that  is  Deliverable  3  from  the  research  task  that  funded  this  report, 
summarises  the  success  factors  for  SoSE.  Table  1,  and  the  others  of  its  ilk  that  appear  below, 
were  developed  from  those  published  in  that  report.  From  reading  the  success  factors,  it  can 
be  seen  that  a  socially-aware  approach  is  needed  that  has  a  spiral  nature  and  that,  during  the 
early  spirals,  the  degree  of  rigour  needs  to  reflect  the  resources  allocated  to  the  task  and  the 
relative  maturity  of  the  constituent  systems  compared  with  the  SoS. 

Section  5  discusses  the  selection  of  approach  for  land  force  capability. 

Table  1.  Success  factors  associated  with  the  selection  of  an  overarching  SoS  approach 

Key  Success  Factors  in  SoS  Engineering 
Overarching  Principles  and  Approach 

1 .  SoSE  can  only  commence  when  a  SoS  has  been  Acknowledged.  Acknowledged  SoS  have  a 
governance  structure,  defined  roles  and  responsibilities  and  resources  allocated  to  the  task.  (OSD, 
2008;  Turner  et  al,  2009). 
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2.  SoSE  is  not  the  same  as  traditional  project-based  SE.  Attempts  to  apply  project-centric  top- 
down  SE  to  SoSE  will  not  produce  good  outcomes  (Norman  and  Kuras,  2006;  Ireland,  2011; 
Dahmann  et  al.,  2011;  Stevens,  2011  &  Kalawsky  et  al,  2012). 

3.  SoSE  requires  a  multi-methodological  approach  with  high  adaptivity.  This  is  because  SoSE 
is  too  complex  to  tackle  with  a  single  worldview  and  a  single  methodology  (Kline,  1 995;  Cook  and 
Allison,  1998). 

4.  It  is  necessary  to  design  a  specific  SoS  practice  to  meet  each  circumstance  as  each  military 
SoS  has  its  own  challenges  and  (OSD,  2008;  Gorod  et  al,  2008,  USAF  2005). 

5.  It  is  essential  to  focus  on  the  design  strategy  and  on  trades  both  when  the  formal  SoS  is  first 
established  and  throughout  the  SoS  evolution  (OSD,  2008). 

6.  Organizational  as  well  as  technical  issues  need  to  be  addressed.  This  applies  to  decisions 
and  SE  trades  and  decisions  (OSD,  2008;  Stevens,  2011). 

7.  Continuous  characterisation  of  the  evolving  SoS  is  needed.  This  would  cover  the  current  and 
forecast  utilisation  environments,  the  status  of  constituent  systems,  and  technology  (Norman  and 
Kuras  2006;  OSD,  2008). 

8.  A  SoSE  Process  that  balances  top-down  and  bottom-up  analysis  and  synthesis  is  essential 

(OSD,  2006;  Stevens,  2011). 

9.  It  is  vital  to  achieve  constituent  system  project  office  buy-in.  It  is  also  necessary  to  maintain  it 
through  rewards  that  incentivise  project  offices  to  deliver  SoS  outcomes  not  just  constituent  system 
outcomes  (Norman  and  Kuras,  2005;  OSD,  2008).  It  is  essential  to  encourage  constituent  projects 
to  take  on  the  attributes  of  a  “good”  constituent  system  project  (Dahmann  and  Baldwin,  2011). 

10.  Metrics  should  be  in  place  to  assess  the  “goodness”  of  key  stakeholder  inputs  to  SoSE. 

This  particularly  applies  to  the  CONOPS  (Turner  et  al,  2009).  The  metrics  should  measure: 

•  Comprehensiveness  of  Stakeholder  identification  and  participation 

•  Stakeholder  understanding  of  the  SoS  problem,  constraints,  solution  trade  space  and 
potential  solution  concepts. 

•  Determination  of  the  most  reasonable  increment  spiral  durations 

•  Ability  of  CONOPS  to  be  achieved  through  existing  enterprise  architecture  structure. 

1 1 .  The  SoSE  framework  should  identify  outcome  spaces  at  multiple  levels  of  scale  and  from 
multiple  points  of  view.  For  example  strategy,  warfighting  concepts,  capability  portfolios, 
projects,  operations  and  sustainment  (OSD,  2008,  Norman  and  Kuras,  2006;  Ireland,  2011). 

12.  SoSE  should  be  incremental  and  evolutionary.  Employ  “spiral  fieldings”,  each  of  which  ends  in 
defined  SoS  capabilities  (Rechtin  1991;  OSD,  2008;  Norman  and  Kuras,  2006;  Turner,  2009; 
Ireland,  2011). 

13.  It  is  important  to  have  a  long-term  roadmap  of  SoS  evolution.  This  will  facilitate  the 
progression  from  short-term  fixes  to  an  integrated-by-design  approach.  Nontheless,  it  is  not 
sensible  to  try  and  define  a  detailed  SoS  state  very  far  into  the  future;  aim  for  a  few  years  hence 
(OSD,  2008). 

14.  The  first  iteration  needs  to  be  investigatory  and  pragmatic.  It  is  unlikely  that  the  SoSE  Team 
will  have  sufficient  information  to  manage  the  iteration  using  project  management  principles, 
indeed  during  this  phase,  it  is  crucial  to  be  pragmatic  about  the  selection  of  process  activities  to  be 
performed  and  SoSE  artefacts  to  be  produced  (Dahmann  et  al,  2011).  It  is  essential  that  the 
artefacts  are  chosen  to  be  informative  and  of  practical  utility  to  the  constituent  program  offices  and 
reflect  the  SoSE  resources  allocated. 

15.  The  initial  integration  of  SoS  is  most  effectively  done  using  integration  enabling 
technologies.  Integration  developments,  “glueware”,  should  be  used  to  graft  elements  together,  ie 
a  bottom-up  approach  has  merit  (Norman  and  Kuras,  2006;  OSD  2008). 

16.  Adopt  a  minimal  set  SoS  integration  principles.  These  are  used  to  steer  projects  towards 
achieving  outcomes  that  will  facilitate  integration  with  the  SoS,  this  approach  is  particularly  useful 
is  the  early  iterations  (Norman  and  Kuras,  2006;  Kalawsky  et  al,  2012). 
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17.  Move  to  an  architecture-informed  SoSE  approach  as  SoSI  capability  matures.  Model-driven 
enterprise  architectures  have  been  found  to  be  effective  in  SoS  coordination  activities  (USN, 
2006a  &  b;  OSD,  2008). 


3.3  Organisational  Roles  and  Responsibilities 

Regardless  of  the  approach  chosen,  it  is  clear  that  to  efficiently  integrate  something  as 
complex  as  the  land  force  capability  SoS  that  will  involve  many  projects,  there  needs  to  be  a 
small  number  of  staff  dedicated  to  specific  SoSE  roles  and  that  there  also  needs  to  be  SoS- 
related  roles  within  the  constituent  system  SPOs.  These  dedicated  roles  and  responsibilities 
are  a  vital  element  of  SoSE  capability,  without  which  progress  is  unlikely  to  occur  and  Lane 
(2009)  provides  guidance  on  how  best  to  size  these  efforts  based  on  around  a  dozen 
quantitative  metrics  used  to  characterise  military  SoS.  The  success  factors  for  the 
Organisational  Roles  and  Responsibilities  Element  are  shown  in  Table  2. 


Table  2.  Success  factors  associated  with  organisational  roles  and  responsibilities 


Key  Success  Factors  in  SoS  Engineering 
Organisational  Roles  and  Responsibilities 

18.  The  systems  engineers  at  the  SoS  level  undertake  different  roles  from  those  at  the 
constituent  system  level  (OSD,  2008). 

19.  Success  depends  on  the  ability  of  SoS  managers  to  work  across  systems  and  balance 
technical  and  nontechnical  issues.  This  requires  experienced,  capable  SoS  managers  and  SE 
teams  (OSD,  2008). 

20.  The  SoS  Systems  Engineering  Team  should  explicitly  articulate  how  the  program  will 
manage  all  requirements  (statutory,  regulatory,  derived,  certification)  (OSD,  2008). 


3.4  Discussion  on  SoS  Engineering  Environments 

Just  as  it  is  inconceivable  to  undertake  systems  engineering  activities  within  a  project  without 
appropriate  tools,  the  complexity  of  the  SoSE  challenge  is  such  that  it  is  vital  to  employ  tools. 
These  tools  need  to  be  selected  to  support  the  approach  chosen  but  as  a  minimum  these  tools 
will  need  to  support  SoS  definition,  architectural  design,  engineering  analysis,  modelling  and 
simulation,  knowledge  management,  and  testing  and  experimentation.  It  is  vital  that  these 
tools  can  provide  value  without  requiring  all  the  detail  of  the  component  systems. 

For  convenience,  we  have  included  the  SoS  Test  and  Evaluation  facilities  and  environments 
into  this  element.  These  would  comprise  a  combination  of  SoS-specific  aspects  and  general 
Defence  engineering  infrastructure. 
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The  success  factors  for  the  SoS  Engineering  Environments  Element  are  shown  in  Table  3. 

Table  3.  Success  factors  associated  with  the  SoS  Engineering  Environments 

Key  Success  Factors  in  SoS  Engineering 
SoS  Engineering  Environments 

21 .  Collaborative  design  and  decision  support  environments  are  essential.  These  are  needed  to 
facilitate  communication  between  warfighters,  SoS  engineering,  design  teams  of  component 
systems,  and  other  stakeholders  (Norman  and  Kuras,  2006;  Stevens,  2011  &  Freeman,  2012). 

22.  Validated  Federations  of  simulations  are  needed  to  perform  SoS  performance  assessment 
and  utility  analysis  (Rumford,  2008  &  Stevens,  2011). 

23.  “Discovery  engineering”  facilities  that  can  support  prototypes,  experiments,  and 
playgrounds  are  essential.  These  are  needed  to  demonstrate  the  utility  of  proposed  constituent 
systems  and  de-risk  SoS  integration  (Norman  and  Kuras,  2006  &  Kalawsky  et  al.  2012). 

24.  Effective  SoSE  and  information  management  tools  are  essential  (OSD,  2008).  These  are 
needed  to  support  and  record  SoS  design  and  design  rationale.  Commonality  of  tools  is,  at 
present,  the  only  effective  way  to  achieve  to  support  the  generation  of  SoS  artefacts  and  reuse. 

25.  T&E  facilities  are  needed  to  assess  the  evolving  performance  of  the  SoS  at  defined 
opportunities. 

26.  Establish  a  SoS  engineering  culture  within  the  capability  development  and  acquisition  community. 


3.5  Discussion  on  SoSE  Personnel 

SoSE  success  relies  on  employing  personnel  possessing  key  SoSE  competencies.  Many  of  these 
competencies  are  in  the  leadership  and  social  skills  areas  but  high-level  SE  technical 
competencies  are  also  needed.  Recent  seminal  work  provides  a  good  insight  into  the 
competencies  required  to  succeed  in  SoSE  engineering  and  management  roles.  The  UK  MoD 
competency  framework  for  "Systems  People"  (Oxenham  and  Swales,  2010)  draws  on 
psychological  research  in  the  UK  to  define  the  types  of  people  needed  for  this  type  of  work, 
whereas  INCOSE  (2010)  provides  a  competency  framework  that  can  be  profiled  for  the 
technical  and  managerial  competencies  that  people  need  to  exhibit  to  deliver  good  SoS 
outcomes.  It  is  recognised  that  these  people  will  be  high  achievers  with  broad,  successful 
career  backgrounds  and  they  can  be  expected  to  have  risen  to  executive  managerial  positions. 
Notwithstanding  this,  there  are  models  from  overseas  that  demonstrate  that  these  people  can 
be  sourced  and  nurtured  successfully. 
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The  success  factors  for  SoSE  personnel  are  given  in  Table  4. 

Table  4.  Success  factors  associated  with  the  SoSE  personnel 

Key  Success  Factors  in  SoS  Engineering 
SoSE  Personnel 

27.  It  is  necessary  to  identify  and  manage  critical  workforce  competencies.  It  is  also  essential  to 
recognise  that  the  competency  and  personal  attribute  profiles  for  SoS  systems  engineers  are 
different  from  that  needed  for  project  systems  engineers  (Turner  et  al,  2009;  Freeman,  2012). 

28.  Innovative  contracting  arrangements  are  needed  to  secure  the  services  of  SoS  SEs  (Cook 
2012,  Unewisse  and  Cook,  2011). 


3.6  Discussion  on  SoS  Training 

It  is  axiomatic  that  an  enduring  capability  will  require  a  well  thought-through  training 

component.  Given  the  environment  in  which  SoSE  will  operate,  and  the  specialised,  complex 

nature  of  the  activity,  several  training  packages  will  be  needed. 

•  Executive  training  for  incoming  SoS  program  office  staff  -  this  training  will  need  to 
cover  SoSE  principles,  roles  and  responsibilities,  introduction  to  SoSE  practices,  and 
the  use  of  the  SoSE  environment. 

•  Model-based  Systems  Engineering  training  for  the  SoSE  processes  and  tools 
employed.  This  would  be  needed  for  several  levels:  supervised  practitioner, 
practitioner,  and  expert. 

•  SoSE  soft  skills  training  to  maximise  the  potential  for  synergistic  collaboration 
between  the  stakeholder  groups. 

The  success  factors  for  SoSE  training  are  given  in  Table  5. 

Table  5.  Success  factors  associated  with  the  SoS  training 

Key  Success  Factors  in  SoS  Engineering 
SoS  Training 

29.  A  structured  formation  path  is  needed  to  develop  SoS  personnel.  (Freeman,  2012;  Cook 
2012). 

30.  SoS  training  will  need  to  include  training  programs  for  all  SoS  stakeholder  groups. 

(Freeman,  2012). 

31.  A  SoS  Engineering  culture  must  be  established  in  which  automatic  consideration  of  the 
SoS  requirement,  implications  and  impact  are  considered  by  all  stakeholders  of  the 
capability  acquisition  and  force  generation  processes  (Blockley  &  Godfrey,  2000).. 
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3.7  SoS  System  Design  and  Synthesis 

Over  the  last  twenty  years,  a  specific  set  of  SoSE  design  and  synthesis  approaches  has  been 

developed  and  assessed.  The  success  factors  for  these  are  given  in  Table  6. 

Table  6.  Success  factors  associated  with  the  SoS  design  and  synthesis 

Key  Success  Factors  in  SoS  Engineering 
SoSE  Design  and  Synthesis 

32.  SoS-level  end-to-end  performance,  behavioural,  and  utility  analyses  are  essential  (OSD, 
2008). 

33.  SoS  solutions  should  be  based  on  open  systems  techniques  and  loose  coupling  (OSD, 
2008;  Norman  and  Kuras,  2006). 

34.  There  is  a  need  for  “glueware”.  The  more  independent  the  constituent  systems  projects  have 
been,  the  greater  the  likely  level  of  glueware  needed  (Norman  and  Kuras,  2006). 

35.  “Discovery  engineering”  (prototypes,  experiments,  playgrounds,  etc.)  is  essential  to 
recognise  “goodness”  in  constituent  systems  (Norman  and  Kuras,  2006). 

36.  SoSE  strategy  needs  to  include  early  and  iterative  implementation  on  a  development 
environment  or  part  of  the  SoS  (OSD,  2008). 

37.  The  SoS  SE  Team  will  need  to  identify  and  disseminate  the  information  architecture.  This 
will  form  the  basis  for  collaboration  and  the  technical  standards  that  will  apply  to  SoS  (e.g.  CIOG, 
2012).  The  SoS  SE  Team  will  also  need  to  develop  a  technical  response  to  achieving  and 
assessing  conformance  to  the  above  (OSD,  2008). 

38.  A  key  activity  in  SoSE  is  understanding  the  nature  of  constituent  systems  and  technical 
interfaces  between  them  (OSD,  2008;  Kalawsky  et  al  2012).  Collecting,  storing  and  maintaining 
this  information  is  key  to  SoS  analysis  and  update  planning  and  implementation.  To  the  greatest 
extent  possible,  this  information  needs  to  be  maintained  by  the  constituent  system  SPOs  because 
the  SoS  PO  will  never  have  the  resources  to  undertake  such  as  large  task. 

39.  SoS  success  hinges  on  succeeding  in  Human  Systems  Integration.  These  needs  to  be 
considered  at  both  the  constituent  system  level  and  the  level  of  the  SoS. 

40.  SoSE  needs  to  co-evolve  with  the  relevant  doctrine  and  standard  operating  procedures  in 
order  to  realise  the  potential  of  the  SoS. 

41 .  Effective  processes  to  capture  lessons  learned  from  force  generation  and  operations  are 
essential  to  ensure  alignment  between  SoSE  and  SoS  user  needs.  This  is  often  stated  by 
rarely  implemented  in  practice. 


3.8  SoS  Test  and  Evaluation 

SoS  T&E  is  now  considered  one  of  the  major  challenges  for  both  the  T&E  community  and  the 
SE  community  (Dahmann  and  Heilmann,  2012).  Building  on  the  experience  of  the  last  20 
years,  SoS  T&E  has  now  identified  the  success  factors  given  in  Table  7. 
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Table  7.  Success  factors  associated  with  the  SoS  test  and  evaluation 

Key  Success  Factors  in  SoS  Engineering 
SoSE  Test  and  Evaluation 

42.  Define  a  best  practices  model  for  SoS  T&E.  SoS  T&E  should  be  used  as  a  continuous 
improvement  process  to  provide  supporting  capabilities  and  limitations  information  for  end  users 
and  feedback  to  SoS  and  System  SE  teams  toward  evolution  of  the  SoS.  (Wilson  et  al,  2011). 

43.  Define  SoS  capability  test  approach.  Rethink  T&E  of  systems  in  an  operational  context  and 
systems  interoperability  away  from  system  testing  toward  integrated  capability  SoS  testing.  (Wilson 
et  al,  201 1). 

44.  Define  characteristics  of  successful  T&E.  Identify  the  process  by  which  we  can  change  and 
influence  the  governance  of  SoS.  Mature  and  improve  templates  to  define  a  minimum  set  of 
characteristics  that  are  required  to  govern  SoS  T&E  efforts.  (Wilson  et  al,  201 1 ). 

45.  Recognise  and  employ  SoS  guidance.  Ensure  that  guidance  or  SoS  SE  is  recognized  and 
employed  on  growing  numbers  of  SoSs.  (Wilson  et  al,  201 1 ). 


3.9  Discussion  on  SoSE  Research  and  Development 

The  Canadian  PRICIE  capability  framework  explicitly  lists  R&D  as  one  of  its  core  headline 
inputs  to  capability.  This  reflects  their  view  that  any  defence  capability  requires  R&D  to 
maintain  long-term  effectiveness.  Given  that  SoSE  is  still  evolving  internationally  and  that 
Australian  Defence  SoSE  practice  is  yet  to  emerge,  there  are  many  research  topics  that  require 
attention.  The  list  below,  consolidated  from  Valerdi  et  al  (2008),  provides  a  consensus  of 
research  topics  produced  during  a  meeting  of  US  commercial,  defence  industry,  and  academic 
cognoscenti: 

•  Determine  how  to  architect  and  evolve  no-single-owner  SoS  through  studying 
successful  approaches  and  innovative  practice.  Specific  issues  include: 

o  Overall  approach  selection 
o  Evolution  and  guided  emergence 
o  Definition,  tailoring,  and  application  of  SoSE  artefacts 
o  Maturation  of  model-based  systems  engineering  approaches 

•  Determine  how  best  to  achieve  SoS  attributes  (SoS  ‘ilities)  such  as  security, 
adaptability,  flexibility,  agility,  scalability,  modularity,  sustainability,  resilience,  net- 
centric  vulnerability. 

•  Determine  how  best  to  identify  and  form  SoS  systems  engineers  and  how  to  extend  the 
human  limits  to  handling  complexity.  Technical  approaches  to  the  latter  would  include 
enhanced  knowledge  management,  data  mining,  automated  reasoning  under  uncertainty 
about  SE  artefacts  such  as  requirements  specifications,  SysML  representations,  etc. 

More  recently,  Rhodes  (2012)  recognised  that  systems-of-systems  research,  because  of  the 
breadth  of  its  contributing  disciplines,  required  synthesis  and  cataloguing  in  a  way  that  makes 
the  literature  and  practice  accessible  to  researchers,  educators  and  the  practitioner 
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community.  Rhodes  through  the  Systems  Engineering  Advancement  Research  Initiative 
(SEAri),  is  gathering  international  collaborators  to  advance  this  work  and  the  first  author  of 
this  report  is  to  become  involved  and  bring  the  fruits  of  this  research  back  to  Australia. 


In  addition,  there  is  a  need  to  effectively  capture  the  SoSE  lessons  and  insights  to  shape  the 

next  cycle  of  the  SoS  development. 

The  success  factors  for  SoSE  research  and  development  are  given  in  Table  7. 

Table  8.  Success  factors  associated  with  the  SoS  research  and  development 

Key  Success  Factors  in  SoS  Engineering 
SoSE  Research  and  Development 

46.  It  is  necessary  to  establish  research  programs  that  can  tackle  the  SoSE  challenge  at 
multiple  timescales.  It  is  necessary  to  develop  approaches,  methods,  tools  and  techniques  that 
can  address  today’s  SoSE  capability  issues  as  well  as  identifying  practices  for  the  medium  and 
long  term  (Freeman,  2012). 

47.  SoSE  is  a  practice-based  discipline  and  evidence  will  be  needed  from  practice  to  validate 
proposed  innovations  in  SoSE  (Boehm,  2012). 
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4.  Towards  an  outline  of  a  Land  SoSE  methodology 

4.1  Background 

There  are  several  schools  of  thought  in  SoSE.  The  first  is  that  SoSE  can  be  achieved  through 
operating  Traditional  (project-based)  Systems  Engineering  (TSE)  at  the  next  level  up  and 
adding  some  additional  processes  and  artefacts.  We  shall  call  this  the  TSE  School.  Indeed  the 
original  work  on  architecture  frameworks  and  the  views  followed  this  paradigm  (Levis  and 
Wagenhals,  2000)  as  did  the  US  Naval  Systems  of  Systems  SE  Guidebook  (USN,  2006  a&b). 

Perhaps  the  antithesis  of  this  is  the  approach  spearheaded  by  the  New  England  Complex 
Systems  Institute  and  Mitre  that  we  will  call  the  Massachusetts  School  that  is  based  on 
complexity  theory.  This  approach  strongly  advocates  that  the  top-down  classical  TSE 
approach  is  not  well  suited  to  SoSE  because  it  cannot  handle  the  complexity  of  SoS  and 
because  the  preconditions  for  TSE  to  be  successful  are  not  evident  in  SoS  (Bar-Yam,  2003; 
Norman,  2005;  Norman  and  Kuras,  2006;  DeRosa  et  al,  2008).  Norman  (2005)  lists  these 
preconditions  as: 

•  The  specific  desired  outcome  must  be  known  a  priori,  and  it  must  be  clear  and 
unambiguous  (implied  in  this  is  that  the  edges  of  the  system,  and  thus  responsibility, 
are  clear  and  known); 

•  There  must  be  a  single,  common  manager  who  is  able  to  make  decisions  about 
allocating  available  resources  to  ensure  completion; 

•  Change  is  introduced  and  managed  centrally; 

•  There  must  be  “fungible”  resources  (that  is  money,  people,  time,  etc.)  which  can  be 
applied  and  reallocated  as  needed. 

In  response  to  the  inappropriateness  of  TSE  for  SoS  problems,  Norman  and  Kuras  (2006) 
formulate  Complex  Systems  Engineering  that  is  described  by  a  number  of  principles,  many  of 
which  can  be  found  in  commercial  practices  and  the  management  literature. 

Yet  another  approach,  which  we  shall  call  the  British  Soft  Systems  School,  is  strongly  motivated 
by  soft  systems  thinking  as  advocated  by  Checkland  and  Scholes  (1999)  and  Hitchins  (2003). 
This  school  focuses  on  achieving  shared  meaning  and  objectives  across  the  stakeholder  group 
and  identifying  different  types  of  SoSE  against  various  perspectives,  namely,  product,  service, 
operational,  enterprise  and  TSE  subscribers,  and  across  Hitchins'  Five  Layers  Model  of 
Systems  Engineering  (INCOSE  UK,  2010).  This  school  has  the  capacity  to  embrace  the  other 
schools  but  what  it  brings  in  breadth  and  philosophical  underpinnings  it  lacks  in  specifics. 

What  we  are  seeking  is  to  define  an  operational  approach  for  land  force  capability 
development  that  draws  on  all  available  approaches  while  meeting  the  constraints  and 
realities  of  the  Australian  defence  situation.  The  current  US  DoD  approach,  derived  from  the 
TSE  School,  offers  this  as  potentially  does  what  we  shall  call  the  European  School  that  is 
described  by  Kalawsky  et  al.  (2012),  which  provides  fresh  ideas  based  on  contract-based  (or 
service-based)  design,  commercial  methodologies,  semantic  modelling  and  simulation,  and 
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upgraded  SoSE  tools.  Given  the  maturity  of  the  US  DoD  approach,  we  will  use  this  as  a  basis 
from  which  to  design  a  land  force  capability  SoS  approach. 

4.2  The  Australian  Defence  Enterprise  Environment 

The  Australian  Defence  Enterprise  environment  is  rather  different  from  that  of  our  allies  and 
this  impacts  strongly  on  the  type  of  SoSE  that  would  work  well  in  this  country.  These 
differences  stem  from  the  fact  that  Defence  SoSE  is  not  established  in  any  formal  way  in 
Australia,  and  this  being  the  case,  the  contemporary  literature  indicates  that  it  would  be 
premature  to  employ  an  architecture-driven  methodology  in  the  first  instance.  In  addition,  the 
Australian  Defence  Enterprise  does  not  possess  the  SoS-level  of  SE  capability  that  is  found  in 
many  overseas  countries  (Unewisse  and  Cook  2011a  &  b)  and  importantly  it  lacks  the 
research,  innovation,  and  education  capability  that  could  readily  support  the  introduction  of 
large-scale,  highly-structured  SoSE  efforts:  these  would  need  to  be  grown  in  conjunction  with 
the  SoSE  capability. 

It  would  be  fair  to  rate  the  Australian  Defence  SoS  Capability  Level  on  the  SEI  CMMI  - 
Acquisition  (SEI,  2010)  scale  as  Level  0  "Incomplete",  which  is  defined  as  a  process  that  either 
is  not  performed  or  is  partially  performed.  The  insight  from  this  statement  is  not  so  much  an 
indictment  of  the  current  situation,  but  a  realisation  that  to  propose  an  elaborate  approach 
would  be  premature.  Rather,  it  would  be  better  to  aim  for  a  maturity  level  of  Level  1  "Initial" 
that  would  employ  flexible  processes,  which  would  need  to  depend  on  the  competencies  of  a 
few  highly  capable  people  as  opposed  to  proven  practices. 

Furthermore,  recent  work1  also  indicates  that  a  substantial,  disciplined  engineering  approach 
would  not  receive  traction  within  Defence  for  resources,  social,  staffing,  and  cultural  reasons 
relating  to  the  lack  of  impact  of  previous  architecture-framework-based  initiatives.  We  also 
know  that  there  is  no  appetite  for  increasing  staff  levels  inside  Defence  and  hence  we  need  an 
approach  which  is  roughly  cost-neutral  compared  with  the  status  quo.  Thus  a  new  synthesis 
of  the  available  approaches  is  essential. 

4.3  A  Starting  Point:  The  US  DoD  Approach 

The  Systems  Engineering  Guide  for  Systems  of  Systems  (OSD,  2008),  enumerates  seven  SoSE 
Elements  (principal  activities),  five  Focus  Areas  for  SoSE  management,  16  processes  and  five 
SoSE  Principles  that  have  found  utility  across  a  range  of  US  SoS  developments.  These  are 
shown  in  Table  7.  This  was  essentially  the  starting  point  for  the  ongoing  evolution  of  the  US 
DoD  approach  to  SoSE.  We  note  this  initial  set  of  concepts  relates  to  the  US  acquisition 
processes  and  the  complementary  T&E  processes  did  not  receive  equal  billing  at  that  time. 
Considerable  further  work  by  the  US  DoD  addressed  this  issue  and  focussed  on 
operationalizing  the  concepts  (Dahmann  and  Heilman,  2012;  Dahmann  2012).  The  outcome  of 
this  work  is  embodied  within  the  wave  representation  of  the  spiral  development  model 
(Dahmann  et  al.,  2011),  which  is  centred  on  orchestrating  the  evolution  of  a  SoS  towards 
defined  milestones,  see  Figure  3. 


1  Interview  analysis  from  the  SOS  PIC  Scoping  Study  and  the  SoS  PIC  Health  Check  Study. 
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Table  9.  Key  Concepts  from  the  Systems  Engineering  Guide  for  Systems  of  Systems  (OSD,  2008) 

Key  Concepts  from  the  Systems  Engineering  Guide  for  Systems  of  Systems 

Core  SoSE  Elements 

1.  Translating  SoS  Capability  Objectives  into  High-Level  SoS  Requirements. 

2.  Understanding  the  Constituent  Systems  and  Their  Relationships  over  Time. 

3.  Assessing  Extent  to  Which  SoS  Performance  Meets  Capability  Objectives  over  Time. 

4.  Developing,  Evolving  and  Maintaining  an  Architecture  for  the  SoS. 

5.  Monitoring  and  Assessing  Potential  Impacts  of  Changes  on  SoS  Performance. 

6.  Addressing  SoS  Requirements  and  Solution  Options. 

7.  Orchestrating  Upgrades  to  SoS. 

SoSE  Principles 

1.  Addressing  organizational  as  well  as  technical  issues  in  making  SE  trades  and  decisions. 

2.  Acknowledging  the  different  roles  of  systems  engineers  at  the  system  versus  the  SoS  level  and 
the  relationship  between  the  SE  done  at  the  two  levels. 

3.  Conducting  balanced  technical  management  of  the  SoS. 

4.  Using  an  architecture  based  on  open  systems  and  loose  coupling. 

5.  Focusing  on  the  design  strategy  and  trades  both  when  the  formal  SoS  is  first  established  and 
throughout  the  SoS  evolution. 

Focus  Areas  for  SoSE  Management 

1.  Program  Requirements:  The  SoS  SEP  should  define  how  the  program  will  manage  all 
requirements  (statutory,  regulatory,  derived,  certification). 

2.  Technical  Staffing  and  Organization  Planning:  The  SEP  should  show  how  the  program  will 
structure  and  organize  the  program  team  to  satisfy  requirements. 

3.  Technical  Baseline  Management:  The  SEP  should  establish  a  technical  baseline  approach. 

4.  Technical  Review  Planning:  The  SEP  should  show  how  the  program  will  manage  the  technical 
effort,  including  the  technical  baselines,  through  event-based  technical  reviews. 

5.  Integration  with  Overall  Management  of  the  Program:  The  SEP  should  link  SE  to  other 
management  efforts,  including  the  Acquisition  Strategy,  test  planning,  sustainment  planning, 
configuration  management,  risk  management,  and  life-cycle  management. 

Relationship  between  SoSE  Core  Elements  and  SoSE  Processes 


Technical  Processes  Technical  Management  Processes 
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Figure  3.  The  SoSE  Wave  Model  (Dahmann  et  ah,  2011) 
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Dahmann  and  Heilmann  (2012)  provide  guidance  on  the  principal  activities  that  a  SoS 
program  office  would  conduct  and  the  key  artefacts  that  US  DoD  would  expect  to  see 
produced  as  listed  in  Table  8  extracted  from  Dahmann  (2012).  Those  artefacts  in  bold  within 
Table  8  are  of  particular  significance  to  SoS  Test  and  Evaluation  activities.  Even  this  carefully 
considered  set  of  artefacts  may  prove  challenging  in  the  Australian  context  that  would  be 
characterised  by  relatively  small  SoSE  teams.  Hence  the  selection  and  definition  of  SoSE 
artefacts  would  be  an  important  activity  during  the  SoSE  initialisation  stage. 

Table  8.  SoSE  Artefacts  produced  in  each  of  the  Wave  Model  steps 


Step  in  Model 

Artifacts 

SoS  Analysis:  Provides 
an  analysis  of  the  "as  is" 
SoS  and  the  basis  for 

SoS  evolution  by 
establishing  an  initial 

SoS  baseline  and 
developing  initial  plans 
for  the  SoS  engineering 
efforts. 

Characterize  SoS 

•  Capability  objectives 

•  SoSCONOPs 

•  Constituent  System  Info 

•  SoS  Technical  Baselines 

•  SoS  Performance  Measures  &Methods 

•  SoS  Performance  Data 

•  SoS  Requirement  Space 

•  SoS  Risks  &  Mitigations  Plan  for  SoS  SE 

•  SE  Planning  Elements 

•  SoS  Master  Plan 

•  Agreements 

Develop  and  Evolve 

SoS  Architecture: 

Develops  and  evolves 
the  persistent  technical 
framework  for 
addressing  SoS 
evolution  and  a 
migration  plan 
identifying  risks  and 
mitigations. 

•  SoS  Architecture 

Plan  SoS  Update: 

Evaluates  the  SoS 
priorities,  options  and 
backlogs  to  define  the 
plan  for  the  next  SoS 
upgrade  cycle. 

•  Allocated  baseline 

•  Risks  and  mitigations 

•  Agreements 

•  Implementation.  Integration  &  Test  Plans 

•  Integrated  Master  Schedule  (IMS) 

•  Updated  Master  Plan.  Technical  Baselines, 
and  Requirements  Space 

Implement  SoS 

Update:  SoS  SE  team 
monitors  systems 
implementation  and  test 
and  leads  SoS 
integration  and  test. 

•  SoS  Test  Report 

•  SoS  Technical  Plans,  Requirements 
Space,  Performance  Data 

•  System  Test  Reports 

•  SoS  IMS 

•  SoS  Technical  Baselines 
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4.4  Adapting  the  US  DoD  approach  to  suit  the  Australian  Context 

4.4.1  Guiding  Principles 

Internationally,  large  architecture-framework-based  approaches  have  not  always  been  found 
to  be  effective  for  reasons  such  as  the  dynamism  of  the  environment  (the  rate  of  change  of 
constituent  systems  exceeds  the  capability  of  the  architecture  team  to  record  it  and  use  it  to 
inform  action);  the  lack  of  ability  for  central  planning  and  design  to  influence  constituent 
systems  in  a  timely  manner;  and  the  sheer  scale  of  the  effort  required  to  capture  and  maintain 
a  Defence  Architecture  Framework  model  using  a  rigorous  approach  (Kalawsky  et  al,  2012). 
The  bottom-up,  equipment-driven  SoSE  approaches,  see  for  example  Norman  and  Kuras 
(2005),  have  found  utility  when  there  are  many  systems  to  integrate  and  little  opportunity  to 
shape  the  development  of  constituent  systems.  Thus,  given  the  environmental  factors  and  the 
fact  that  SoSE  is  yet  to  be  formalised  within  Defence,  the  US  DoD  approach  will  need  to  be 
tailored  carefully. 

We  suggest  that  it  would  be  best  to  start  from  the  position  that  the  task  of  land  force  capability 
SoSE  activity  in  the  short  term  is  to  maximise  the  integration  of  current  and  planned  land 
capabilities  with  as  little  impact  on  the  constituent  systems  as  possible. 

Morris  et  al  (2006)  provides  a  useful  conceptual  framework  for  an  appropriate  approach: 

"Collaborative  system-of-sys terns  governance  involves  abandoning  the  notion  of 
rigid  top-down  governance  of  ...  processes,  standards,  and  procedures  and 
adopting  peer-to-peer  approaches.  Such  collaborative  system-of-systems 
governance  is  clearly  at  odds  with  the  natural  tendency  of  business  and  military 
organizations,  because  it  means  that  the  "chain  of  command"  must  evolve  to  a 
"web  of  shared  interest."  Collaborative  system-of-systems  governance  requires 
cooperation  between  separate  authorities,  even  when  there  is  no  formal 
agreement." 

Figure  4  illustrates  the  key  players  in  an  Australian  Defence  SoSE  context.  Imperial  projects 
are  those  that,  by  their  very  scale  and  influence,  shape  many  other  projects  and  existing 
capabilities.  Examples  include  Land  400  Land  Combat  Vehicle  System,  Air  6000  Air  Combat 
Capability,  AIR  5077  Airborne  Early  Warning  and  Control  System,  SEA  4000  Air  Warfare 
Destroyer,  AIR  7000  Maritime  Patrol  Aircraft  Replacement.  Component  projects  provide 
specific  capabilities  such  as  sensing  systems,  weapons  systems  and  training  systems.  In 
contrast,  "glue"  projects  are  enabling  projects  that  provide  the  enabling  infrastructure  for 
C4ISREW  such  as  communication  links,  command  support  systems  and  network 
management.  It  is  envisaged  that  the  SoS  Team  would  comprise  a  few  individuals  that  would 
be  tasked  with  the  roles  and  responsibilities  described  below.  The  arrows  on  Figure  4  show 
lines  of  influence  and  agreement,  with  the  size  of  the  arrow  indicating  the  relative  complexity 
of  the  link.  These  entities  work  within  an  SoSE  engineering  environment  that  is  informed  by 
the  constraints  listed  earlier,  and  by  the  efforts  of  organisational  elements  within  Defence 
tasked  with  achieving  network  integration  (e.g.  the  Land  Network  Integration  Centre, 
(Rawlinson,  2011),  the  Chief  Information  Officer  Group,  The  Tactical  Information  Exchange 
Integration  Office,  Defence  Systems  Integration  Technical  Advisory,  etc.). 
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It  is  important  to  appreciate  that  the  projects  each  have  their  own  asynchronous  lifecycles 
with  the  "glue"  projects  potentially  being  the  shortest,  extending  up  to  the  imperial  projects 
such  as  military  vehicles  that  span  decades. 

In  practise,  the  overall  SoSE  Team  is  likely  to  be  a  distributed  construct  and  could  comprise 
members  from  one  or  more  of  the  SoSE  entities  enumerated  in  Figure  4.  The  first  ("1"  in 
Figure  4)  would  be  an  operational  capability-based,  acknowledged  SoSE  Team,  for  example, 
joint  amphibious  capability  or  joint  fires.  Although  these  dedicated  SoSE  Teams  may  have 
considerable  authority  and  influence,  particularly  should  they  be  in  direct  support  to  a  cross¬ 
project  Program  Management  Steering  Group  (PMSG)  oversighting  issues  across  multiple 
projects,  they  will  tend  to  be  small  and  are  unlikely  to  directly  control  funds.  Note  that  current 
PMSGs  are  not,  in  general,  supported  by  a  SoSE  Team. 
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Figure  4.  Key  players  in  Australian  Defence  SoSE 

SoSE  Teams  can  also  be  located  within  an  "imperial  project"  ("2"  in  Figure  4).  These  teams 
have  the  advantage  of  direct  authority  within  the  large-scale  project  and,  through  the 
dominant  role  of  the  imperial  project,  plus  access  to  comparatively  high  levels  of  funding,  are 
able  to  shape  the  associated  projects  into  a  SoS.  A  major  limitation  of  these  SoSE  Teams  is  the 
constraint  placed  on  them  to  limit  their  scope  to  the  immediate  focus  on  the  project. 

Finally,  SoSE  Teams  will  also  be  found  within  the  major  'glue'  projects.  This  is  an  almost 
inevitable  consequence  of  the  nature  of  these  projects,  as  they  provide  the  interfaces  between 
the  other  projects  and  also  have  the  most  holistic  approach  to  the  overall  SoS.  As  a 
consequence,  there  is  a  tendency  for  the  other  projects  to  transfer  much  of  their  SoS 
integration  responsibilities  and  risks  to  the  'glue'  projects.  Unfortunately,  the  'glue'  projects 
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are  rarely  funded  to  address  the  scope  of  the  broader  integration  challenges  transferred  to 
them  by  the  rest  of  the  constituent  SoS  projects. 

A  relatively  recent  development  is  the  emergence  of  very  senior  PMSGs  up  to  the  3-star  level 
that  have  the  authority  to  direct  individual  projects  and  redirect  funds.  However,  this  high- 
level  governance  function  tends  to  be  unsupported  by  the  sort  of  significant  SoSE  support  that 
could  be  provided  by  a  dedicated  SoSE  Team.  A  possible  approach  would  be  to  establish  a 
small  dedicated  SoSE  team  to  support  the  senior  PMSG,  and  then  coordinate  the  efforts  of  the 
distributed  (and  hopefully  augmented)  SoSE  Teams  within  the  "glue"  and/or  "imperial" 
projects. 

4.4.2  Review  of  success  factors 

Below  is  a  summary  of  the  SoSE  success  factors  that  are  salient  to  the  design  of  a  nascent  SoSE 
approach  to  suit  the  land  force  capability  challenge. 

•  SoS  function  needs  to  be  formally  acknowledged,  resourced  and  established. 

•  The  first  activity  will  be  to  identify  the  capability  needs  in  the  form  of  an  agreed 
capability  objectives  and  concept  of  operations. 

•  The  approach  will  have  to  be  designed  to  suit  the  need  (in  full  appreciation  that  it  will 
initially  be  investigatory  and  pragmatic)  where: 

o  The  activities  will  be  continuous  and  iterative. 

o  The  initial  iteration  will  be  largely  bottom-up  and  driven  by  the  in-service  or 
contracted  project  systems. 

o  Resource  will  be  modest  -  the  SoS  Team  will  be  small, 
o  A  minimal  set  of  relatively  simple  artefacts  are  used  for  the  initial  iteration, 
o  SoS  Team  activities  are  informed  by  a  minimal  set  of  rules  of  the  games  rather 
than  process. 

•  Constituent  system  SPOs  will  need  to  be  tasked  to  provide  (and  keep  configuration 
managed)  salient  project  information  to  the  SoS  Team. 

•  The  key  SoSE  artefact  for  the  Land  SoSE  activities  will  be  the  Agreements  between  the 
SoS  Team  and  the  constituent  system  SPOs. 

•  The  approach  needs  to  be  socially  aware,  have  SoS  Teams  work  across  systems  and 
balance  technical  and  non-technical  issues. 

•  The  approach  will  need  to  be  compatible  with  the  SoSE  environment  available. 

Furthermore,  TTCP  (2011)  strongly  advocates  that  one  of  the  principal  lessons  learned  over 
recent  years  in  defence  SoS  practice  is  that  it  is  imperative  that  the  SoSE  effort  is  aligned  with 
resources.  Resources  are  allocated  to  projects  and  as  such  the  constituent  project  SPOs  need  to 
take  on  as  many  tasks  as  possible,  such  as  the  provision  of  SoS-relevant  configuration- 
managed  project  information,  the  establishment  and  execution  of  SoSE  agreements,  and  SoS 
compatibility  de-risking.  The  SoS  Team  is  unlikely  ever  to  have  the  resources  to  collect  and 
maintain  the  project  information  or  conduct  extensive  analysis  or  testing. 

Given  the  current  land  force  integration  state  (Rawlinson,  2011),  it  is  a  reasonable  assertion 
that  the  initial  land  force  capability  SoS  iteration  would  be  based  on  the  application  of  SoS 
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enabling  technologies  ("glueware")  to  achieve  interoperability  between  in-service  capabilities 
and  those  soon  to  enter  service.  The  provision  of  the  materiel  to  achieve  the  integration  will  be 
provided  by  what  we  have  termed  "glue"  projects  in  Figure  4.  Thus  the  design  of  the  SoSE 
approach  needs  to  address  how  the  SoS  integration  principles  and  success  factors  would  be 
implemented  if  a  glue  project  were  the  main  driving  force  for  SoS  integration. 

In  summary,  the  success  factors  coupled  with  the  environmental  realities  discussed  earlier 
indicate  the  need  to  adopt  a  lightweight  approach  that  would  establish  a  small  SoSE  Team 
comprising  highly  competent  people  that  could  operate  effectively  with  minimal  formal 
structure,  informal  co-ordination  meetings,  lightweight  processes,  and  minimal  infrastructure. 

4.4.3  SoS  Activities  during  Early  Iterations 

4.4. 3.1  SoS  Team  Initialisation 

SoSE  activities  commence  when  a  SoS  has  been  acknowledged,  resources  are  allocated,  and 
the  SoSE  Team  is  appointed.  The  first  task  of  the  SoS  Team  would  be  to  establish  the 
boundary  of  the  SoS,  its  context,  the  stakeholder  group  and  the  capability  needs  it  is  being 
assembled  to  fulfil.  This  work  would  result  in  the  Capability  Objectives  artefact  that  would 
describe  the  capabilities  needed  by  the  user,  ideally  based  on  some  definitive  or  authoritative 
materials,  but  interpreted  through  workshops  and  war  games.  In  contrast  to  conventional 
project  systems  engineering,  the  SoSE  Team  would  have  the  ongoing  role  of  translating 
capability  needs  into  technical  requirements  and  identifying  new  needs  as  the  situation 
changes  and  the  SoS  evolves.  (After  initialisation,  the  latter  activity  is  bundled  with  SoS 
Analysis,  see  below.) 

The  SoSE  Team  would  work  with  the  key  military  stakeholders  to  establish  how  constituent 
systems  in  the  SoS  will  be  employed  in  an  operational  setting  and  with  SPOs  to  define  the 
functionality  expected  from  each  constituent  system.  Mission  thread  analysis  workshops  or 
similar  (Wood  et  al,  2011)  have  proven  effective  in  producing  this  information.  The  results 
from  this  activity  would  be  recorded  in  a  SoS  Concepts  of  Operations  (SoS  CONOPS). 

At  this  stage,  initial  Constituent  System  Information  comprising  technical  and  programmatic 
information  is  provided  to  the  SoSE  Team  by  the  SPOs  providing  the  constituent  systems. 
SoS-level  Risks  and  Mitigations  are  also  identified  and  the  tracking  and  updating  processes 
initiated. 

SoS  T&E  Planning  commences  at  this  stage  and  is  refined  during  the  SoS  Analysis  activity. 
Dahmann  and  Heilmann  (2012),  OSD  (2008),  among  others,  note  that  comprehensive  OT&E  of 
a  substantial  system  is  impractical  and  hence  significant  thought  needs  to  be  given  to 
addressing  the  SoS  T&E  issues  that  are  enumerated  in  Section  6.2  and  in  the  consequent 
planning  of  a  meaningful  T&E  program. 

During  the  initialisation  phase,  the  first  version  of  the  Agreements  between  the  SoS 
participants  (the  four  groups  shown  in  Figure  4)  should  be  drafted.  The  Agreements  formalize 
roles  and  responsibilities  of  SoS  participants  at  a  broad  level  and  record  organizational 
relationships,  roles,  and  responsibilities,  and  specific  commitments  of  participants  in  a 
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development  increment.  It  is  recognised  that  because  SoS  cut  across  organizational 
boundaries,  agreements  are  critical  to  SoSE  success  (Dahmann,  2012).  It  is  critical  that  the 
relevant  SoS  Plan  and  T&E  elements  be  accepted  by  the  constituent  projects  and  the  other 
relevant  agencies  (e.g.  doctrine  developers)  as  most  of  the  implementation  will  need  to  be 
undertaken  by  these  SoS  participants. 

4.4.3. 2  SoS  Analysis 

During  SoS  Analysis,  the  SoS  systems  engineer  would  be  responsible  for  working  with  the  SoS 
manager  and  other  stakeholders  to  take  the  Capability  Objectives,  SoS  CONOPS  and  System 
Information  and  translate  the  capability  needs  statements  into  key  SoS-level  Requirements 
that  can  inform  the  production  of  a  SoS  Technical  Baseline,  SoS  Performance  Data,  and  SoS 
Performance  Measures  and  Methods.  As  the  understanding  of  the  SoS  matures,  the  technical 
baseline  would  expand  to  contain  analogues  of  many  conventional  SE  artefacts  such  as  a 
functional  decomposition  to  constituent  systems,  functional  flow  definition  in  the  form  of  a 
mission  thread  analysis,  performance  allocation,  etc.  In  order  to  produce  the  artefacts  listed  it 
would  be  necessary  to  perform  a  performance  allocation  between  constituent  systems.  It  must 
be  appreciated  that,  in  the  first  instance,  this  could  not  be  done  with  any  precision  but  the  aim 
would  be  to  improve  this  and  other  aspects  of  the  SoS  Analysis  with  each  iteration.  The 
information  produced  during  the  analysis  provides  the  foundation  for  the  technical  planning 
needed  to  evolve  the  capability  over  time  and  artefacts  such  as  the  SoS  Planning  Elements,  the 
SoS  Master  Plan.  These  need  not  be  detailed  and  could  take  the  form  of  a  roadmap.  Updated 
Agreements  between  the  stakeholder  communities  could  also  be  used  to  document  mutual 
responsibilities  between  the  stakeholders. 

The  SoS  analysis  also  needs  to  assess  the  level  of  SoS  integration  readiness  of  the  constituent 
systems  and  then  identify  the  level  of  system  integration  required  from  each  constituent 
system.  This  will  establish  some  of  the  SoS  integration  risks  as  well  as  identifying  both 
requirements  and  risks  that  need  to  be  addressed  by  the  constituent  SPOs.  Such  analysis  is 
complicated  by  the  asynchronous  development  and  delivery  of  the  constituent  project  as  well 
as  new  projects  and  systems  being  introduced.  This  requires  the  SoS  planning  to  be  able  to 
evolve  over  time,  shaping  and  incorporating  additional  systems  and  projects  relevant  to  the 
ongoing  development  of  the  SoS. 

4.4.3. 3  SoS  Architecture  Development 

The  SoS  Architecture  defines  the  way  the  constituent  systems  work  together  and  would 
include  the  functional  analysis  artefacts  mentioned  earlier  that  would  describe  the  end-to-end 
functionality,  data  flow  definition,  functional  interfaces,  and  communications.  The  SoS 
Architecture  would  need  to  map  the  SoS  functional  needs  onto  current  and  forthcoming 
constituent  systems  in  order  to  support  performance  allocation  and  gap  analysis. 

It  is  important  for  SoS  engineers  to  appreciate  that  the  purpose  of  this  activity  is  to  obtain 
answers  to  specific  short-term  questions  such  as  "What  are  the  functional  gaps?"  or  "What 
capabilities  might  need  to  be  enhanced?",  not  to  produce  a  comprehensive  architecture 
framework  description  per  se.  Thus  the  approaches  used  to  record  and  reason  about  the 
architecture  need  to  be  matched  to  the  questions  to  be  answered  and  the  resources  available. 

As  a  consequence,  specific  architecture  products  should  be  carefully  and  sparingly  selected  to 
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deliver  the  required  answers  and  insights.  Nevertheless,  there  are  likely  to  be  a  large  number 
of  SoS  artefacts  and  architecture  products.  Where  possible,  the  SoSE  Team  should  seek  to 
engage  the  relevant  constituent  SPOs  to  generate  many  of  the  artefacts.  The  small  SoS  Team 
should  focus  on  the  major  whole-of-capability  artefacts  and  shaping  and  synthesising  the 
outputs  of  the  constituent  projects. 

4.4.3. 4  SoS  Update  Planning 

Armed  with  the  fruits  of  the  analysis  and  architecture  development  activities,  the  next  activity 
is  to  identify  those  needs  and  gaps  to  be  addressed  in  the  next  iteration  cycle.  A  systems 
engineering  design  cycle  then  follows  to  identify  the  preferred  solution  for  addressing  the 
selected  capability  gaps.  This  leads  to  updated  baselines  and  other  artefacts  mentioned  earlier. 
Implementation,  Integration  and  Test  Plans,  and  an  Integrated  Master  Schedule.  New 
projects  are  then  established  to  produce  or  modify  constituent  systems  to  implement  in 
accordance  with  these  plans.  These  plans  and  schedules  need  to  be  agreed  by  the  constituent 
SPOs  and,  as  relevant,  incorporated  into  their  own  internal  plans  and  schedules. 

Note  that  longer-term  issues  and  gaps  can  be  captured  in  a  lower  resolution  SoS  roadmap. 

4.4. 3.5  SoS  Update  Implementation 

This  activity  monitors  the  implementations  that  are  being  managed  by  the  SPOs  at  the 
constituent  system  level  and  compares  them  against  the  SoS  Integrated  Master  Schedule,  the 
Risk  and  Mitigation  Plan  and  the  known  SoS  dependencies.  In  parallel,  the  SoS  team  plans 
and  then  conducts  SoS-level  testing  and  evaluation  in  accordance  with  the  plans  and  produces 
a  SoS  Test  and  Evaluation  Report.  The  activity  continues  on  to  update  the  as-is  technical 
baseline  and  other  artefacts  to  reflect  the  evolving  state  of  the  SoS. 

4.5  Addressing  SoS  Challenges  from  a  Constituent  System  SPOs 
Perspective 

Dahmann  and  Baldwin  (2011)  state  that  little  attention  has  been  paid  to  how  systems 
engineers  within  SPOs  should  address  the  engineering  of  new  constituent  systems  to  enable 
them  to  support  current  and  future  SoS.  Notwithstanding  this,  they  have  found  through  a 
search  of  contemporary  US  capability  development  and  acquisition  instructions  that  there  are 
several  points  where  DoD  policy  or  guidance  calls  for  the  SoS  context  to  be  addressed.  These 
are  illustrated  against  the  development  lifecycle  in  Figure  5. 

Dahmann  and  Baldwin  (2011)  go  on  to  state  that  new  US  policy  on  development  planning  has 
instituted  a  policy  shift  from  a  platform  focus  to  a  capability  focus,  and  this  change  requires 
consideration  of  the  context,  particularly  project  interdependencies,  at  the  Materiel 
Development  Decision  for  a  new  acquisition  program.  They  go  on  to  say  that  current  US 
acquisition  policy  also  emphasizes  the  SoS  context  in  key  programmatic  and  technical  plans 
including  the  acquisition  strategy,  systems  engineering  plan,  and  test  and  evaluation  plan  and 
that  these  are  reviewed  at  major  milestones.  Furthermore,  they  state  that  US  SE  guidance 
emphasizes  the  SoS  context  as  part  of  program  formulation,  requirements,  and  integration 
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and  that  key  SE  technical  reviews  include  considerations  of  interdependencies,  interfaces,  and 
information  exchange  requirements  at  the  events  shown  in  Figure  6. 
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Figure  6.  US  DoD  Technical  reviews  that  address  the  SoS  context 


As  with  many  aspects  of  policy,  Dahmann  and  Baldwin  (2011)  recognise  that  there  remain 
unanswered  questions  about  how  best  to  ensure  that  the  intent  of  the  policy  and  guidance  can 
be  thoughtfully  enacted  in  practice. 

The  approach  for  integrating  largely  pre-existing  systems  advocated  by  the  Massachusetts 
School,  in  particular  Norman  and  Kuras  (2006),  address  this  issue  by  advocating  the  use  of 
reward  mechanisms  to  incentivise  SPOs  to  develop  their  products  and  services  in  such  a  way 
as  to  maximise  SoS  outcomes.  This  has  been  found  to  be  successful,  but  it  will  necessitate  a 
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refocus  of  project  priorities  and  the  definition  of  projects'  outcomes  both  at  the  constituent 
system  level  and  at  the  SoS  level.  This  would  become  a  significant  driver  for  shaping  the 
agreements  established  between  the  SoSE  Team  and  the  SPOs. 

Similarly,  Dahmann  and  Baldwin  (2011)  discuss  "good"  behaviours  to  be  encouraged  within 
constituent  system  SPOS.  These  include: 

•  Understanding  the  role  in  the  larger  SoS  capability,  and  considering  this  role  when 
capturing  requirements  and  designing  the  constituent  system  T&E  plan; 

•  Devising  designs  that  facilitate  functionality  changes  and  extensions  and  that  possess 
adaptable  interfaces; 

•  Employing  short  development  cycles  to  provide  more  opportunities  for  aligning 
constituent  system  properties  with  SoS  needs; 

•  Recognising  that  the  SoS  Team  is  a  user  of  the  project  configuration  management  and 
requirements  processes  and  tools  and  designing  them  to  welcome  outside  inputs;  and 

•  Employing  a  business  model,  which  provides  a  way  to  support  changes  and  address 
issues  of  maintenance. 

In  the  short  term,  this  is  likely  to  require  active  participation  by  SoSE  Team  members  in 
constituent  project  IPTs.  Given  the  small  size  of  the  SoSE  Team,  this  will  constrain  both  the 
scale  and  number  of  SoSE  efforts  that  can  be  initially  pursued. 

Over  time,  a  combination  of  efforts  including  systems  engineering  training  that  incorporates 
SoS  issues,  implementation  of  high-level  reporting  and  reward  mechanisms  for  SoS  risks  and 
issues  within  projects,  and  cultural  changes  within  the  capability  development,  will  increase 
the  level  of  awareness  and  willingness  to  address  SoS  engineering  and  integration  issues 
within  the  SPOs.  As  SoSE  becomes  a  more  accepted  and  ingrained  element  of  the  SPOs,  the 
level  of  effort  needed  by  the  SoSE  team  in  each  project  will  decrease,  allowing  for  increases  in 
the  scope  and  scale  of  the  SoSs  addressed  -  without  large  increases  in  personnel. 

4.6  Requirements  for  a  Land  SoS  support  environment 

First  and  foremost,  the  environment  needed  to  support  Land  SoS  development  and  evolution 
must  achieve  information  integration.  It  must  be  able  to  import,  catalogue  and  mine  data  and 
knowledge  in  whatever  forms  it  is  represented.  At  present,  this  represents  a  serious  challenge 
because  of  the  disparate  information  systems  employed  across  Defence  and  the  relative 
immaturity  of  the  tools  available  to  perform  this  function  at  the  SoS  level.  The  advent  of  the 
rollout  of  the  DSTO  Whole  System  Analytical  Framework  heralds  an  opportunity  to  capture 
project  definition,  planning  scenarios,  requirements  and  test  concepts  in  coherent  machine- 
readable  form,  and  this  represents  a  key  enabler  for  a  successful  SoSE  environment  (Robinson, 
2010),  one  that  can  transcend  the  contractual  boundary  (Do  et  al.,  2011).  Furthermore,  the 
latest  research  in  MBSE  environments  demonstrates  promise  in  the  integration  of  tool  suites 
through  bi-directional  semantic  translators  between  tool-specific  representations  and  provides 
a  comprehensive,  operationalized  MBSE  environment  that  spans  behaviour  and  relational 
modelling  through  to  simulation  of  the  final  design  implementation  (Kalawsky  et  al.,  2012b). 
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In  the  interim,  conventional  SE  tools  and  knowledge  management  technology  can  be 
employed  to  good  effect  but  it  must  be  understood  that  the  capability  of  the  tools  employed 
will  drive  the  nature  of  the  information  collected  and  the  level  of  abstraction  selected.  Note 
that  implementing  such  MBSE  tools  will  require  increased  investment  in  system  engineering 
training  as  well  as  training  in  the  MBSE  tools  themselves  in  order  to  realise  their  potential  to 
support  SoS  integration. 

4.7  Wider  factors  that  shape  the  proposed  Land  SoS  approach 

As  discussed  in  the  section  on  the  components  of  SoS  capability,  SoSE  is  a  socio-technical 
challenge,  and  there  are  many  social  and  cultural  issues  to  address  to  transition  from  the 
entrenched,  stovepiped  approach  to  one  that  embraces  that  project  outcomes  and  performance 
metrics,  particularly  those  for  communications  and  information  system  "glue"  projects,  need 
to  reflect  predominantly  SoS  metrics.  It  is  something  of  an  axiom  in  the  SoSE  research 
community  (IEEE  2012)  that  SoSE  is  an  intensely  social  activity,  which  presents  many  cultural, 
behavioural,  ethical  and  organisational  challenges  to  the  inherently  single-project  acquisition 
ethos  predominant  in  both  the  defence  and  civil  sectors.  For  example,  see  Flenshaw  (2012). 
This  is  will  need  an  active  program  to  evolve  the  organisational  culture  towards  a  SoSE 
approach.  This  will  involve  a  phased  approach  over  a  number  of  years  to  both  grow  the  SoSE 
capabilities  and  to  establish  the  acceptance  of  SoSE  within  Defence  as  an  effective  way  to 
realise  future  integrated  warfighting  capabilities. 
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5.  Towards  Requirements  and  Specification  Practices  for 
Land  Force  Capability  Integration 

5.1  The  Type  of  the  Requirements  Information  Needed  for  SoSE 

Firstly,  from  the  foregoing,  it  is  our  belief  that  requirements  and  specification  practices  for 
land  force  capability  integration  will  be  strongly  shaped  by  the  overall  SoSE  approach 
adopted.  In  the  tailored  US  DoD  approach  described  above,  requirements  appear  both  at  the 
SoS  level  and  also  exist  for  each  constituent  system. 

At  the  land  force  capability  level,  Dahmann  et  al.,  (2010)  state  that  SoS  requirements  explicitly 
do  not  take  the  form  of  requirements  for  a  Major  Capital  Equipment  (MCE)  project;  that  is  a 
large  set  (often  thousands)  of  detailed  operational  requirements  or  specified  technical 
requirements. 

Rather  SoS  requirements  are  captured  by  three  artefacts:  the  SoS  Capability  Objectives,  the 
SoS  Concept  of  Operations  (CONOPS),  and  the  SoS  Requirements  Space.  A  description  of 
these  artefacts  largely  due  to  Dahmann  et  al.  (2010)  appears  below. 

SoS  Capability  Objectives  are  a  statement  of  top-level  objectives  for  the  SoS.  They  describe  the 
capabilities  needed  by  the  user,  ideally  based  on  some  definitive  or  authoritative  materials 
(e.g..  White  Paper,  Defence  Capability  Planning  Guidance,  strategies  or  Australian  Capability 
Context  Scenarios).  They  are  used  by  the  SoS  Team  and  stakeholders  as  the  foundation  for  SoS 
requirements,  metrics,  etc.  The  capability  objectives  provide  a  basis  for  translating  operational 
needs  into  high-level  requirements,  assessing  performance  to  capability  objectives,  and 
developing  an  architecture  and  solution  options. 

The  SoS  CONOPS  describes  how  the  functionality  of  the  constituent  systems  in  the  SoS  will  be 
employed  in  an  operational  setting.  The  CONOPS  is  developed  by  the  prospective  operational 
users  and  with  active  participation  from  the  SoS  Team  to  describe  the  way  users  plan  to 
operate  and  use  constituent  systems  to  achieve  the  objectives,  as  influenced  by  the  various 
environments  and  conditions  anticipated.  The  CONOPS  is  developed  in  parallel  with  the 
capability  objectives  and,  as  the  capability  objectives  evolve,  the  CONOPS  should  evolve  in 
detail,  as  well.  The  SoS  Team  and  the  SPOs  use  the  CONOPS  to  define  the  SoS  requirements 
space,  to  identify  aspects  of  constituent  systems,  which  could  impact  the  SoS  design,  and  to 
select  performance  metrics  and  test  environments. 

The  SoS  Requirements  Space  bounds  the  first-order  SoS  user  needs  (including  operational 
tasks  and  missions)  and  defines  the  functions  required  to  provide  the  capability  across  the 
range  of  user  environments  likely  to  be  encountered.  It  should  be  noted  that  this  artefact  is  a 
'requirements  space'  versus  a  set  of  'requirements',  because  in  a  SoS,  'requirements'  are  taken 
on  by  the  constituent  systems  to  meet  the  SoS.  The  requirements  space  includes  both  the  SoS 
Capability  Backlog  and  Problem  Reports.  It  is  developed  by  the  SoSE  Team  in  liaison  with 
constituent  system  SE  teams  and  with  strong  engagement  with  SoS  and  constituent  system 
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operational  communities.  It  is  used  by  the  SoSE  Team  and  constituent  system  SE  teams  to: 
determine  information  needed  to  understand  systems  and  relationships,  compare 
performance  to  capability  objectives,  develop  an  SoS  architecture,  identify  areas  to  be 
addressed  in  an  increment(s),  identify  solution  options,  develop  a  plan  for  SoS  increment(s), 
and  develop  a  plan  to  test  and  evaluate  the  changes. 

It  can  be  seen  that  the  three  artefacts  record  information  at  a  different  level  of  abstraction  from 
the  Function  and  Performance  Specification  that  is  the  principal  requirements  document  used 
with  the  Australian  Defence  Capability  Development  process.  Defence  (2009),  states  that  the 
FPS  specifies  the  formal  requirements  for  the  Materiel  System  and  provides  the  basis  for 
design  and  qualification  testing  of  the  system.  The  FPS  provides  the  vehicle  for  the  capture  of 
formal,  verifiable  and  unambiguous  requirements,  effectively  'distilled'  from  the  OCD.  The 
FPS  is  intentionally  written  using  formal  language,  with  all  requirements  in  the  FPS  traceable 
to  needs  identified  in  the  OCD.  The  FPS  becomes  part  of  the  contract  between  the  Defence 
Materiel  Organisation  and  the  supplier.  In  contrast,  the  SoS  Requirements  Space  is  an 
interpretation  of  the  SoS  Capability  Objectives  and  the  SoS  CONOPS  by  the  SoS  Team  that  is 
used  for  the  purposes  mentioned  above  and  to  shape  the  Agreements  between  the  SoS  Team 
and  the  constituent  system  SPOs.  These  agreements  must  facilitate  both  shaping  the 
component  projects  (SPOs)  to  take  into  account  the  SoS  requirements  and  shaping  the  SoS 
artefacts  to  take  into  account  key  drivers  and  constraints  from  the  constituent  projects. 

We  see  that  the  most  effective  way  of  coordinating  and  managing  the  relationships  necessary 
for  a  good  SoSE  outcome  is  through  the  Agreements.  The  definition  of  Agreements  provided 
in  Dahmann  et  al.  (2010)  states:  The  Agreements  formalize  roles  and  responsibilities  of  SoS 
participants  at  a  broad  level  (e.g.  Charter)  as  well  as  specific  commitments  of  participants  in  a 
development  increment.  Because  SoS  cut  across  organizational  boundaries,  agreements  are 
critical  to  SoS  SE  success.  SoS  and  system  management  and  SE  teams  (and  contracting  officers 
and  commercial  contractor  representatives,  as  needed)  define  agreements  among  participants 
regarding  organizational  relationships,  roles,  and  responsibilities,  and  to  manage  interactions 
of  participants  and  other  stakeholders.  The  SoSE  Team  should  also  facilitate  appropriate 
agreements  between  the  SoS  constituent  projects. 

Recent  research  points  to  a  richer  use  of  agreements  to  achieve  engineering  outcomes  in  a  SoS 
context.  Kalawsky  et  al  (2012a)  proposes  that  commercial  ideas  can  be  brought  to  bear  to 
tackle  the  vexing  problem  of  integrating  constituent  systems  into  SoS  through  what  is  termed 
contract-based  design  where  the  contract  encapsulates  the  conditions  for  correct  integration  as 
opposed  to  constituent  system  requirements  and  constraints.  Quinton  et  al.  (2010)  state  that 
contract  and  interface  frameworks  are  emerging  as  the  formalism  of  choice  for  system  designs 
that  require  large  and  dispersed  teams,  or  where  the  supply  chain  is  complex.  They  go  on  to 
say  that  contracts  are  usually  expressed  as  pairs  of  assumptions,  or  properties  that  the 
environment  must  satisfy,  and  guarantees,  the  properties  that  must  be  satisfied  by  each 
particular  component.  Ben-Hafaiedh  et  al.  (2010)  increases  the  sophistication  of  the 
contracting  relationship  by  adding  interfaces  to  make  a  3-tuple. 

Contract-based  design  is  centred  on  forming  SoS  by  integrating  pre-existing,  well-defined 
components.  Initially  these  were  software  components  but  the  technique  is  migrating  to 
hardware  also.  Contract-based  design  uses  formal  mathematical  foundations  for  contract 
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clauses  and  as  such  can  support  proofs  and  other  forms  of  reasoning  as  well  as  automatic 
design.  For  example,  if  the  assumptions  of  each  component  are  contained  in  guarantees 
offered  by  constituent  systems,  then  the  SoS  composition  is  termed  "well  formed"  and  will 
meet  its  specification.  Contract-based  design  is  seen  as  a  major  component  of  the  European 
Commission  research  project  'Designing  for  Adaptability  and  evolutioN  in  System-of-systems 
Engineering  (DANSE)  and  Kalawsky  et  al.  (2012)  show  how  the  contracts  can  be  coupled  to 
design  and  modelling  and  simulation  tools  to  create  and  verify  SoS  compositions. 

Similarly,  Holt  et  al  (2012)  take  a  rich  view  of  requirements  that  suits  the  needs  of  SoSE  well  in 
that  they  consider  requirements  to  comprise  business  requirements,  functional  requirements 
and  non-functional  requirements  (often  termed  constraints).  The  latter  two  categories  are 
normally  included  in  a  project  function  and  performance  specification,  whereas  the  business 
requirements  are  absent  in  that  document,  although  aspects  are  included  in  other  project 
documentation  such  as  the  Operational  Concept  Description  and  the  Acquisition  Business 
Case.  Holt  et  al.  show  how  the  Systems  Modelling  Language  (SysML)  can  be  used  to  capture 
comprehensive  requirements  in  hierarchical  requirements  diagrams,  activity  diagrams, 
sequence  diagrams,  state  machine  diagrams,  block  definition  diagrams,  and  use  cases.  This  is 
a  rich  way  to  record  contracts  and  SysML  has  been  shown  to  be  able  to  be  used  for  automatic 
design,  functional  modelling  and  simulation,  detailed  implementation  and  prototyping 
(Kalawsky  et  al.,  2012a). 

While  the  contract-based  approach  is  yet  to  mature,  the  idea  of  setting  up  an  agreement  to 
supply  (largely  pre-existing)  constituent  systems  based  on  high-level  definitions  of  services 
and  behaviours  has  much  to  commend  it  over  the  approach  that  expects  thousands  of  detailed 
performance  parameters  to  be  known  and  quantitative  performance  thresholds  defined  before 
work  commences.  Indeed,  this  is  how  one  buys  complex  items  such  as  computers,  smart 
phones,  and  cars  in  the  civil  sector.  Such  agreements,  if  coupled  with  appropriate  reward  and 
penalty  mechanisms,  would  facilitate  parallel  action  between  the  SoS  constituent  SPOs,  and 
with  the  SoSE  Team.  However,  care  needs  to  be  taken  that  the  agreements  incorporate 
sufficient  flexibility  to  ensure  that  the  parties  to  these  agreements  are  able  to  evolve  along 
with  the  evolving  nature  of  an  enduring  SoS. 

If  such  agreements  are  put  in  place  to  support  SoS  engineering  and  integration,  then  the 
maturity  of  the  web  of  agreements  between  the  SoS  constituent  SPOs  and  with  the  SoSE  Team 
becomes  a  key  metric  for  effective  SoS  integration  processes. 

5.2  An  Evolving  Requirements  Practice  for  Land  Forces  Capability 

Management  of  enduring  SoS  such  as  the  land  force  capability  would  expect  to  cycle  through 
many  iterations  of  the  Wave  Model.  In  the  early  iterations,  many  of  the  constituent  systems 
would  either  be  in  service  or  already  specified  and  contracted  against  a  detailed  specification. 
In  cases  such  as  these,  the  work  of  the  SoSE  team  is  to  understand  how  the  SoS  will  be  used 
and  to  identify  gaps,  interfacing  difficulties  and  performance  shortfalls,  and  to  work  with  the 
constituent  system  SPO  to  effect  small  changes  to  meet  the  objectives  of  the  SoS  Milestones. 

As  the  SoSE  capability  matures,  greater  value  arises  from  increasing  the  degree  of  top-down 
SoS  design  until  a  middle-out  approach  is  reached,  which  fully  combines  top-down  strategic 
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drivers  with  bottom-up  equipment  and  project  realities  that  is  driven  by  functions,  often  best 
expressed  as  mission  threads.  In  common  with  strategic  planning  and  systems  engineering,  in 
general,  it  is  useful  to  start  with  the  end  in  mind  when  undertaking  SoSE.  Thus,  it  is  proposed 
that,  even  in  early  SoS  iterations,  SoS  artefacts  are  produced,  however  embryonic.  Foremost  of 
these  will  be  the  three  SoS  artefacts  that  define  the  intent  of  the  SoS:  the  SoS  Capability 
Objectives,  the  SoS  CONOPS,  and  the  SoS  Requirements  Space.  Initially  these  could  take  the 
form  of  short  documents  or  posters  even  but,  over  time,  they  should  migrate  to  something  akin 
to  the  Whole-Systems  Analytical  Framework  (WSAF)  (Robinson  et  ah,  2010).  While  WSAF 
was  originally  conceived  for  large  projects,  the  projects  to  which  it  has  been  applied  are  SoS- 
type  glue  projects,  for  example  Land  19  Phase  7,  Ground  Based  Air  Missile  Defence,  and  Land 
400,  the  Land  Combat  Vehicle  System  (LCVS),  the  Army’s  largest,  most  expensive  and  most 
complex  major  capability  equipment  project  to  date,  that  aims  to  deliver  the  mounted,  close- 
combat  capability  to  the  land  force  from  2025.  In  Robinson’s  words: 

“As  we  know,  the  LAND  400  “model”  is  more  than  a  series  of  traceable  information.  It 
includes  additional  information  that  provides  a  greater  level  of  context  and  justification  for 
the  systems  requirements  and  operational  needs  that  is  verging  on  “design  rationale”.  Beyond 
“just”  an  enhanced  traceability  matrix  and  what  should  be  an  improved  OCD  and  FPS  (over 
the  document-centric  alternatives),  what  return  on  investment  does  this  bring?  In  my  view, 
this  MBSE  approach  brings  a  return  in  investment  through: 

•  Extending  SE  practices  to  the  enterprise  level 

•  Improved  completeness  of  capability  definition 

•  Greater  stakeholder  engagement  through  effective  visualisation 

•  Traceability  of  designs 

•  Capture  of  [SoS]  design  rationale 

•  Good  knowledge  management  through  focussing  on  knowledge  not  documents 

•  Simple  knowledge  dissemination  -  accessible  by  stakeholder  groups 

•  Improved  risk  identification  and  management.” 

If  high-level  ownership  of  the  SoS  Capability  Objectives,  the  SoS  CONOPS,  and  the  SoS 
Requirements  Space  artefacts  can  be  achieved,  then  these  artefacts  can  support  the  governance 
and  decision-making  of  the  SoS.  In  particular,  it  will  allow  comparison  of  dissimilar  system 
and  project  proposals  based  on  their  impacts  to  the  SoS,  resulting  in  more  informed  decision¬ 
making  to  deliver  the  broader  SoS  capability  rather  than  consideration  of  project  proposals  in 
isolation. 

Once  SoSE  tools  reach  maturity  and  are  populated,  then  top-down  design  becomes  possible 
and  the  SoS  conceptual  model  can  inform  constituent  system  requirements.  (Flanigan  and 
Brouse  (2012)  propose  a  SoSE  requirements  allocation  process  based  on  traditional  systems 
engineering  that  could  be  considered.)  Until  that  time,  the  best  avenue  to  influence  projects 
and  formalise  the  expectation  of  SoS  needs  and  the  contributions  that  fielded  constituent 
systems  would  be  expected  to  make  to  SoS,  would  be  the  Agreements  between  the  SoSE  Team 
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and  the  SPOs  and,  where  appropriate.  Agreements  between  the  other  parties  shown  in 
Figure  4. 

We  are  proposing  that  the  scope  of  the  SoS  Agreements  proposed  by  Dahmann  et  al.  (2010)  be 
expanded  to  include: 

•  Role  and  responsibilities  of  both  parties; 

•  Specific  commitment  of  both  parties  to  a  SoS  development  increment  including 
coordination  meetings,  demonstrations  or  other  de-risking  activities,  constituent 
system  project  events,  etc; 

•  Information  to  be  provided  from  the  SoSE  Team,  dissemination  mechanisms  and 
timeliness  expectations  (assumptions  in  contract-based  design  language); 

•  Mission  functions  and  behaviours  to  be  delivered  by  the  constituent  system 
(properties  in  contract-based  design  language); 

•  Interface  definition  at  a  high-level  (interface  definition  in  contract-based  design 
language); 

•  How  the  functions  and  interfaces  will  be  verified,  e.g.  demonstration  on  a  SoS 
integration  test  bed; 

•  Information  to  be  provided  by  the  constituent  system  team  and  timeliness 
expectations;  and 

•  Overarching  drivers  for  and  frameworks  of  agreements  for  the  SoS  to  be  endorsed  by 
senior  leadership  /  governance  processes. 

5.3  Multi-Scalar  SoSE 

Land  force  capability  comprises  multiple  battlefield  operating  systems  (BOS),  which  are  SoS, 
or  sub-SoS  of  the  broader  land  force.  This  creates  the  opportunity  for  intermediate  SoSE  teams 
that  build  on  the  existing  BOS  capabilities  such  as  fires,  logistics,  engineering,  and  ISR,  and 
the  established  organisational  and  cultural  frameworks  that  currently  seek  to  address  SoS  in 
the  land  force.  Such  BOS-based  SoSE  teams  should  be  able  to  be  quickly  established  as  they 
are  both  focused  at  a  manageable  level  of  complexity  and  build  on  established  stakeholder 
communities  across  the  relevant  projects  and  land  force  organisations.  The  focused  nature  of 
the  teams  should  also  assist  in  relatively  rapidly  addressing  the  SoSE  challenges  of  each  BOS. 

SoSE  is  still  required  at  the  land  force  level  to  facilitate  the  integration  of  the  intermediate  BOS 
SoS  into  larger  force-level  constructs.  SoSE  teams  at  this  level  would  seek  to  shape  the 
synthesis  of  the  BOSs  into  combined-arms  teams  and  undertake  trade-offs  between  the 
potentially  locally  optimised  BOS  solutions  to  deliver  more  effective  land  force  capability. 

Such  a  multi-scalar  approach  to  land  SoSE  should  both  enable  a  more  rapid  adoption  of  the 
SoSE  approach  in  land  and  reduce  the  risks  of  trying  to  solve  all  of  the  SoS  integration 
challenge  in  one  monolithic  SoSE  effort.  This  would  enable  a  phased  implementation  of  SoSE 
allowing  for  growth  of  land  SoSE  team  capacity  and  for  an  incremental  learning  approach  to 
SoSE  in  land  with  later  SoS  teams  learning  from  the  experience  of  earlier  efforts.  Finally,  such 
a  multi-scalar  approach  would  align  with  the  distributed  nature  of  the  land  SoS  integration 
effort  between  multiple  projects  and  the  land  force  generation  process. 
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Note  that,  although  C4I  can  be  regarded  as  another  BOS  capability,  the  reality  is  that  these 
"glue"  systems  are  ubiquitous  within  and  between  all  the  BOS  SoS.  As  such,  they  are  often 
best  considered  as  a  core  enabler  of  the  overall  land  force  SoS.  This  will  tend  to  result  in  a 
close  relationship  between  these  "glue"  capabilities  as  a  sub-SoS  and  the  force-level  SoSE 
team. 
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6.  T&E  processes  for  Land  Force  Capability  integration 

6.1  Challenges  for  SoS  T&E 

SoS  T&E  is  not  only  perplexing  because  of  the  complexity  of  the  SoS  under  consideration,  but 
also  because  of  how  SoS  are  composed:  they  are  formed  from  constituent  systems,  almost  all 
of  which  are  in  different  stages  of  their  system  lifecycle.  Wilson  et  al,  (2010)  lists  the  SoS  T&E 
challenges  identified  by  the  NDIA  in  2009;  the  ones  that  relate  to  the  Australian  land  force 
capability  SoS  are  as  follows: 

•  Requirements:  If  'requirements'  are  not  clearly  specified  up  front  for  a  SoS,  what  is  the 
basis  for  T&E  of  an  SoS? 

•  Metrics:  What  is  the  relationship  between  SoS  metrics  and  T&E  objectives? 

•  Systems  Changes:  Are  expected  cumulative  impacts  of  systems  changes  on  SoS 
performance  the  same  as  SoS  performance  objectives? 

•  End-to-End  Testing:  How  do  you  test  the  contribution  of  a  system  to  the  end-to-end 
SoS  performance  in  the  absence  of  other  SoS  elements  critical  to  the  SoS  results?  What 
if  the  constituent  systems  are  all  implemented  to  their  specification  but  the  overall  SoS 
expected  changes  cannot  be  verified? 

The  NDIA  have  been  working  on  how  to  address  these  issues,  and  the  solutions  relevant  to 
the  Australian  circumstances  have  been  assimilated  into  Section  7.3. 

6.2  SoS  Questions  to  Answer 

The  following  is  a  list  of  questions  posed  by  Hess  (2010)  that  relate  to  SoS  T&E: 

•  How  much  testing  is  enough? 

•  How  long  will  testing  take? 

•  How  much  will  it  cost? 

•  How  do  I  test  effectively  given  the  compressed  schedule  of  a  "Rapid  Acquisition" 
program? 

•  How  do  I  measure  the  quality  of  my  tests? 

•  What  are  the  most  valuable  tests  for  my  system? 

•  How  should  I  prioritize  my  tests? 

•  How  do  I  make  sure  my  tests  are  representative  of  the  operational  environment? 

•  How  do  I  get  more  knowledge  for  my  dollar? 

•  What  are  the  unique  challenges  in  testing  Systems-of-Systems  (SoS)? 

•  How  do  I  test  a  SoS  without  explicit  requirements? 

•  How  does  my  system  affect  the  SoS  in  which  it  operates? 

•  What  are  the  most  valuable  tests  for  my  SoS? 
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Additional  questions  that  might  be  considered  also  include: 

•  How  can  the  relevant  SoS  T&E  requirements  be  addressed  within  the  T&E  program 
for  each  of  the  constituent  projects? 

•  What  T&E  can  be  undertaken  within  a  constituent  project  to  access  its  potential 
contribution  of  the  larger  SoS(s)? 

•  How  will  T&E  of  the  SoS  shape  future  phases  of  the  constituent  projects? 

Given  the  challenges  inherent  in  SoS  T&E  listed  above,  the  answers  to  these  questions  will 
need  to  be  found  both  in  the  overall  framework  for  SoS  T&E  and  through  formation  of  SoS 
T&E  practitioners  who  can  cope  with  the  inevitable  compromises  and  incompleteness  of  SoS 
T&E. 

6.3  SoS  T&E  Framework 

Dahmann  (2012)  and  Dahmann  et  al  (2010)  provide  a  progress  report  on  the  work  underway 
by  the  NDIA  SE  Division  SoS  SE  and  T&E  Committees  on  best  practices  in  SoS  T&E.  They 
state  that  it  is  important  to  note  that  SoS  T&E  is  fundamentally  different  in  goals,  character, 
and  intent  from  system-acquisition  T&E  that  seeks  to  inform  acceptance  and  deployment 
decisions. 

Dahmann  et  al.  (2010),  Dahmann  and  Heilmann  (2012),  and  Dahmann  (2012)  raise  several 
important  points  to  consider  when  discussing  SoSE  and  T&E.  Firstly,  the  point  is  made  that 
SoSE  and  T&E  is  done  as  an  integrated  part  of  the  SoS  evolution  process  and  SoSE  and  SoS 
T&E  share  key  common  elements.  In  that  context,  SoS  T&E  needs  to  be  focussed  on  SoS 
operational  performance.  Secondly,  it  is  essentially  impossible  to  test  a  substantial  SoS,  so 
other  means  of  evaluating  the  operational  effectiveness  and  suitability  of  SoS  need  to  take  a 
larger  place  on  the  palette  of  T&E  approaches.  These  would  include: 

•  Modelling  and  simulation 

•  Analysis 

•  Experimentation 

•  Exercises 

•  Operations 

•  Incremental  testing 

The  work  of  Dahmann,  Wilson  and  colleagues  invokes  the  goal  of  T&E  from  the  US  DoD 
Acquisition  Guidebook:  "The  overall  goal  of  T&E  is  to  reduce  risk  by  providing  crucial 
information  to  decision  makers"  and  proposes  a  T&E  approach  that  reflects  this  and  is 
characterised  by  the  following  core  ideas: 

•  Integrate  T&E  with  SE  throughout  the  evolution  of  an  SoS; 

•  Focus  T&E  on  risk,  both  in  the  planning  of  the  SoS  and  in  its  implementation; 

•  Employ  a  variety  of  sources  of  evidence  including  prior  T&E  results,  data  from 
analysis,  and  SoS  test  events  as  needed; 

•  Radically  change  how  we  look  at  testing  given  the  growing  prevalence  of  SoS: 
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o  Concepts  of  Developmental  Test  (DT)  and  Operational  Test  (OT)  don't  really 
fit, 

o  Inefficient  to  address  systems  in  operational  SoS  environment  on  a  system-by¬ 
system  basis  (OT  today), 

o  Continue  to  test  individual  systems  to  assess  whether  we  have  developed 
what  we  asked  for,  and 

o  Create  a  new  approach  to  OT  through  cross-systems  support  for  testing 
integrated  capabilities2; 

•  Characterize  SoS  T&E  as  continuous  improvement,  document  the  approach  and  share 
with  the  community; 

•  The  SoSE  Team  needs  to  work  very  closely  with  the  constituent  system  SPOs  in  order 
to  facilitate  a  workable  SoS  solution; 

•  Establish  incentives  of  constituent  systems  to  collaborate  and  achieve  SoS  performance 
objectives;  and 

•  In  later  iterations,  map  SoS  capabilities  and  performance  objectives  to  constituent 
systems  (under  configuration  control). 

The  following  sections  discuss  the  SoS  T&E  activities  that  would  be  conducted  in  each  stage  of 
the  Wave  Model.  It  should  be  noted  that  we  are  proposing  that  the  Agreement  between  the 
SoS  Team  and  the  constituent  system  SPOs  be  expanded  to  include  the  high-level  technical 
performance  and  behaviour  that  the  constituent  system  is  to  provide  to  the  SoS.  This  needs  to 
shape  the  constituent  system  T&E  program  to  access  the  capability  of  the  constituent  system 
to  contribute  to  the  SoS  in  an  operational  context. 

Furthermore,  it  is  important  to  appreciate  SoSE  and  SoS  T&E  are  inherently  evolutionary 
activities  that  progressively  shape  the  SoS  towards  its  goals.  Each  iteration  of  the  wave  model 
provides  an  opportunity  to  shape  the  constituent  systems  to  contribute  more  effectively  to  SoS 
operational  performance.  It  needs  to  be  understood  that  it  may  take  more  than  one  cycle  to 
shape  some  constituent  systems  depending  on  the  synchronisation  of  the  individual  SPO 
capability  cycles  with  the  SoS  wave. 

6.4  SoS  T&E  Activities 

6.4.1  SoS  Initiation 

Dahmann  and  Heilmann  (2012)  assert  that  Capability  Objectives  and  the  SoS  CONOPS 
provide  the  foundation  for  T&E,  providing  SoS  high-level  capability  measures  of  effectiveness 
and  operational  context.  These  are  first  recorded  in  this  stage  and,  together  with  the  associated 
mission  threads  and  military  tasks,  these  form  the  inputs  for  structuring  T&E  activities  and  for 
teasing  out  critical  operational  issues  and  T&E  measures  such  as  SoS  measures  of 
effectiveness,  measures  of  suitability,  and  measures  of  performance. 

At  this  stage,  the  constituent  system  information  is  gathered  and  T&E  results  from  constituent 
system  programs  are  scrutinised  for  information  that  will  impact  on  the  SoS  measures. 

2  This  is  already  being  done  in  Australia  as  demonstrated  through  the  Land  200  trials  that  treated  a  collection 
of  C4I  projects  (Land  75,  Land  125,  JP  2072)  as  an  integrated  SoS. 
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SoS  T&E  planning  commences  at  this  stage  and  is  refined  during  the  SoS  analysis  stage.  This 
should  include  identifying  opportunities  to  incorporate  relevant  SoS  T&E  elements  in 
constituent  SPOs. 

6.4.2  SoS  Analysis 

The  SoS  Capability  and  Performance  Objectives  that  are  updated  during  SoS  Analysis  provide 
the  foundation  for  SoS  T&E.  Acknowledged  SoS  comprise  existing,  fielded  systems,  and  hence 
understanding  current  and  potential  SoS  performance  requires  an  understanding  and 
assessment  of  the  T&E  results  of  constituent  systems,  the  existing  SoS  plus  a  forward  vision 
and  concept  for  the  SoS.  Dahmann  (2012)  states  that  systematic  development  and  analysis  of 
this  data  is  core  to  SoS  analysis  and  since  SoS  typically  brings  systems  together  in  new  ways, 
there  is  a  need  to  obtain  more  data.  This  data  may  be  from  specific  testing,  demonstrations, 
exercises,  modelling  and  simulation,  experimentation  or  analysis.  This  will  include  analysis  of 
the  future  force  operational  effectiveness  using  combinations  of  wargaming,  modelling  and 
simulation. 

The  analysis  should  identify  major  changes  to  the  existing  SoS  and  its  constituent  systems  and 
how  they  may  result  in  change  to  the  SoS,  its  operation  and  performance.  Significant  changes 
to  the  technologies  of  key  constituent  systems  (often  in  combination  such  as  information 
systems,  networking  and  sensors)  may  result  in  major  changes  to  the  scope  of  the  SoS 
capability,  the  SoS  business  processes.  These  changes  may  require  significant  adjustment  in 
SoS  T&E. 

6.4.3  SoS  Architecture  Development  and  Evolution 

Dahmann  (2012)  and  Dahmann  and  Heilmann  (2012)  state  that  T&E  data  on  the  attributes  and 
performance  of  constituent  systems  is  key  to  the  identification  and  analysis  of  architecture 
approaches.  Furthermore,  Dahmann  states  that  because  the  components  of  the  SoS  typically 
include  existing  systems,  T&E  methods  and  tools  can  contribute  to  the  assessment  of 
alternative  architectures  through  the  application  of  various  approaches  including  live,  virtual 
and  constructive  environments  to  assess  alternatives  against  desired  architecture  objectives. 

In  addition,  knowledge  about  T&E  considerations  of  the  constituent  systems  test  and 
certification  requirements  can  inform  architectural  options.  For  example,  if  rapid  changes  are 
required,  then  the  T&E  community  can  help  designers  understand  which  constituent  systems 
would  have  short-enough  time  test  and  certification  lead  times  to  contribute  to  the  solution. 

Conversely,  systems  and  SoS  architectures  can  be  used  to  shape  and  inform  the  design  of  the 
SoS  T&E.  In  addition,  the  SoS  architectures  can  be  used  to  identify  T&E  to  access  the 
performance  of  the  constituent  systems  and  the  ability  to  contribute  to  the  SoS  capability. 
These  constituent  system  T&E  insights  can  in  turn  be  used  to  shape  the  T&E  to  be  undertaken 
within  the  individual  SPOs  to  enable  them  to,  in  part,  assess  the  potential  of  the  systems  to 
contribute  to  the  SoS. 
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6.4.4  SoS  Update  Planning 

Dahmann  (2012)  states  that  T&E  planning  is  a  core  part  of  the  planning  for  an  update,  and, 
given  the  obstacles  to  conducting  full  SoS  testing  with  each  change  in  a  constituent  system 
within  the  SoS,  this  step  is  critical  in  identifying  areas  of  risk  that  warrant  T&E  attention. 
During  this  stage,  expected  changes  in  constituent  systems  are  identified:  those  changes 
planned  by  the  constituent  system  SPO  for  their  own  purposes,  and  those  being  planned  to 
address  SoS  needs.  The  T&E  people  assess  the  potential  impact  of  these  changes  using  the 
architecture  as  the  technical  framework  to  determine  how  the  planned  change  will  affect  other 
constituent  systems  in  the  SoS  given  the  CONOPS  and  system  dependencies. 

Where  risks  are  identified,  the  SoS  T&E  people  will  need  to  determine  how  these  will  be 
assessed  and  how  they  can  be  mitigated.  It  should  be  noted  that  changes  with  no  impact  on 
other  parts  of  the  SoS  (e.g.,  improved  quality  of  sensor  feeds  but  no  change  in  format, 
interfaces,  volume,  etc.)  may  not  need  SoS  attention  beyond  T&E  at  the  system  level. 

6.4.5  SoS  Update  Implementation 

Dahmann  (2012)  asserts  that  T&E  is  a  key  part  of  implementation  for  both  SOS  and 
constituent  systems.  Systems  making  updates  conduct  T&E  at  the  system  level  in  the  normal 
way.  Based  on  the  assessment  of  impacts  of  changes  on  other  parts  of  the  SoS,  added  T&E 
elements  have  been  included  in  the  system  test  plans.  SoS  T&E  includes  monitoring  the 
implementation  of  the  planned  system  testing,  conducting  added  testing  to  address  SoS  risks, 
evaluating  the  results,  and  recommending  changes  in  plans  as  needed.  SoS  capability  results 
are  identified  (both  planned  and  unplanned). 

T&E  questions  surrounding  the  SoS  implementation  step  include  the  following: 

•  What  is  the  operational  performance  of  the  SoS  at  this  increment? 

•  Does  performance  meet  expectations  for  this  increment? 

•  What  are  the  causes  for  any  shortfall  or  improved  performance  relative  to  the  earlier 
SoS  iteration  and  the  expected  SoS  performance? 

•  What  are  the  potential  impacts  of  any  shortfall  or  differences  on  the  next  increment? 

•  What  are  the  risks? 

•  What  evidence  is  there  that  these  changes  will  need  to  be  regression  tested  in  the  next 
increment? 

The  answers  to  these  questions  become  the  basis  for  capabilities  and  limitations  information 
provided  to  end  users  and  serve  as  feedback  to  the  SE  and  T&E  teams  for  the  SoS  and  systems 
toward  continued  evolution  of  the  SoS. 
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7.  Metrics  for  Land  Capability  SoSE  Success 

One  of  the  challenges  associated  with  determining  the  effectiveness  of  any  SE  effort  is  that  it  is 
very  difficult  to  conduct  a  set  of  controlled  experiments  that  can  yield  statistically  significant 
results  as  would  be  done  in  many  areas  of  research  endeavour  such  as  medicine. 

The  preferred  approach  to  date  is  to  conduct  case  study  research.  Cook  (2000)  addressed  the 
problem  of  how  best  to  achieve  SE  success  through  a  literature  survey  that  drew  on 
substantial  published  experience  of  SE  lessons  learned.  Honour  (2004,  2006,  2010a  &  b),  in 
contrast,  formulated  a  pioneering  research  approach  that  interpreted  the  extent  to  which  good 
SE  practices  have  been  followed  in  a  project  and  the  quality  of  those  practices  in  order  to 
establish  the  value  of  systems  engineering  and  the  return  on  investment  for  systems 
engineering.  Honour's  methodology  involves  data  collection  through  confidential  interviews 
with  leading  systems  engineers  and  subsequent  statistical  analysis  of  the  data  set.  Survey 
work  by  Elm  et  al  (2008)  supports  Honour's  results  and  further  surveys  are  being  conducted 
to  enhance  the  results.  These  approaches  have  succeeded  on  the  basis  of  having  a  statistically 
significant  data  set. 

Given  the  lack  of  such  a  data  set,  perhaps  the  most  promising  technique  would  be  to 
benchmark  the  land  SoS  challenge  against  similar  efforts  overseas  suitably  normalised  for  size 
and  level  of  SoS  effort  relative  to  estimated  effort  for  good  practice  (see  Lane,  2009).  Another 
useful  approach  would  be  to  compare  the  evolving  land  SoS  practice  against  the  patterns  for 
SE  success  developed  by  Rebovich  and  De  Rosa  (2011).  A  third  approach  would  be  to  try  to 
identify  SoS  leading  indicators  from  the  work  of  Roedler  et  al  (2010). 

Guckert  (2012)  describes  the  US  Army  SoSE  Organisation  and  a  recent  workshop  surfaced  a 
list  of  metrics  that  apply  to  that  organisation.  However,  stakeholder  comments  suggest  that 
the  percentage-based  metrics  proposed  were  not  very  informative  and  trends  would  be  more 
useful,  costs  should  be  included,  measures  for  suitability  and  effectiveness  need  to  be 
considered,  as  should  metrics  related  to  stakeholder  engagement.  It  was  also  suggested  that  a 
more  holistic  view  of  SoSE  success  could  be  achieved  through  the  use  of  Balanced  Scorecard 
methods.  The  final  version  of  these  metrics  could  well  be  informative  for  the  Australian  Army 
circumstances. 

Agreement  between  the  constituent  projects  and  with  the  SoSE  team  have  been  proposed  in 
this  report  as  a  key  component  of  achieving  effective  SoSE.  A  mathematical  formalism  could 
then  be  established  to  assess  the  overall  'health'  of  the  SoSE  process  based  on  the  measures 
relating  to  the  SoS  agreement  framework. 

Note  that  SoS  analysis  and  T&E  will  identify  issues,  gaps  and  risk  before  and  after  each  SoSE 
wave.  This  information  might  be  used  as  a  method  to  monitor  SoSE  performance. 

Ultimately,  the  success  of  SoSE  will  be  judged  in  terms  of  the  achieved  SoS  capability 
performance  versus  the  cost  of  undertaking  SoSE.  This  could  be  assessed  in  a  manner 
analogous  to  Honour's  approach  that  established,  through  research  covering  over  50  projects, 
that  there  is  a  substantial  return  on  investment  in  performing  systems  engineering.  A  key 
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return  on  investment  is  likely  to  be  a  transfer  of  the  SoS  integration  risks  from  the  operational 
level  (i.e.  the  warfighters),  where  they  have  been  traditionally  addressed,  to  the  capability 
development  domain. 


40 


UNCLASSIFIED 


UNCLASSIFIED 


DSTO-TR-2743 


8.  Conclusion 


This  report  set  out  to  provide  advice  on: 

1 .  Capability/SoS  project  requirements  development  and  specification;  and 

2.  Test  and  Evaluation  (T&E)  process  for  capability/SoS  integration  to  ensure  that  the 
delivered  constituent  systems  will  be  effective  components  of  land  force  capability. 

It  quickly  became  apparent  during  the  research  program  that,  for  the  advice  to  have  meaning 
to  potential  SoS  stakeholders  in  Defence,  it  would  be  necessary  to  place  it  within  the  context  of 
a  SoSE  approach.  This  reports  draws  on  significant  international  literature  and  personal 
communications  with  leading  practitioners  to  present  an  approach  which  is  considered 
"culturally  feasible",  to  use  Checkland's  Soft  System  Methodology  Language  (Checkland  & 
Scholes,  1999),  in  the  Australian  Defence  context.  To  that  end,  this  report  has  provided  an 
outline  of  an  overarching  approach  that  has  the  potential  to  address  the  current  and 
forthcoming  land  SoS  integration  challenge  in  a  way  that  is  very  cost  effective  and  sensitive  to 
the  extant  cultural,  financial  and  organisational  realities  in  which  the  approach  would  be 
employed. 

The  approach  seeks  to  encourage  constituent  system  SPOs  to  embrace  the  land  SoS  vision  and 
keep  it  front-of-mind  in  the  delivery  and  sustainment  of  the  constituent  systems  that  will 
provide  the  equipment  that  will  comprise  the  SoS.  It  will  require  some  modest  additional  roles 
and  responsibilities  in  each  SPO  and  a  small  SoS  Team  to  be  successful. 

In  response  to  the  first  question,  we  show  how  capability  requirements  for  the  SoS  are  elicited 
and  how  the  SoSE  analysis  stage  identifies  how  these  can  be  mapped  onto  constituent  systems 
in  the  form  of  high-level  service  definitions  (analogous  to  service  contracts).  In  the  first 
iteration,  the  SoS  Team  will  have  little  opportunity  in  the  short  tenn  to  change  constituent 
systems  requirements  or  project  scope,  and  hence  the  approach  chosen  to  interface  to  the  SPOs 
is  to  establish  agreements  that  embody  the  service  definitions.  It  is  these  service  and  interface 
definitions  that  are  tracked  by  both  parties  and  form  the  basis  for  guiding  the  evolution  of  the 
SoS. 

In  response  to  the  second  question,  we  assert  that  SoS  T&E  is  fundamentally  different  in  goals, 
character,  and  intent  from  system-acquisition  T&E  that  seeks  to  inform  acceptance  and 
deployment  decisions.  SoS  T&E  is  focussed  more  on  the  operational  performance  of  the  SoS 
because  it  is  deemed  impractical  to  conduct  comprehensive  testing  of  an  evolving  SoS.  Thus 
SoS  T&E  activities  focus  on  the  impacts  that  constituent  systems  iterations  will  make  on  the 
SoS  performance  and  behaviour  outcomes.  As  such,  it  involves  far  greater  use  of  modelling, 
simulation  and  analysis  than  project  T&E,  and  the  conceptualisation  of  training,  exercises,  and 
operations  as  T&E  opportunities.  Nonetheless,  the  SoS  T&E  staff  would  monitor  constituent 
system  project  progress  to  evaluate  whether  the  service  provision  clauses  of  the  agreements 
will  be  achieved. 
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A  final  point  is  that  SoSE  is  quite  different  from  project  SE,  and  SoS  T&E  is  fundamentally 
different  from  project-based  T&E.  Both  disciplines  will  require  highly  capable  and  adaptable 
people  that  display  the  appropriate  mind-set,  attitude  and  aptitude  needed  for  SoS  success. 
Identifying  and  nurturing  the  development  of  these  people  is  an  important  subject  for  further 
consideration. 
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Annex  A:  Definition  of  Terms 

A.l  Introduction 

There  are  many  definitions  for  SoSE  and  what  it  is  often  called  in  Australia  Systems  of  Systems 
Integration.  As  part  of  the  RPDE  QL  62  task,  the  team3  that  included  the  first  author  of  this 
report  was  asked  to  produce  a  list  of  definitions  in  order  to  ensure  a  common  language  and  to 
enable  the  team  to  communication  effectively  with  stakeholders,  we  were  asked  to  define  the 
following  terms. 

•  System  of  Systems  (SoS), 

•  System  of  Systems  Integration  (SoSI);  and 

•  High  End  Systems  Integration  (HESI) 

The  US  definition  of  SoSE  is  also  included  in  the  background  material. 

The  definitions  are  based  on  existing  work  abstracted  in  Section  5.  They  were  discussed  at  the 
Industry  Workshop  held  in  Brisbane  on  24-25  August  2010  with  a  group  of  over  30  RPDE 
representatives  from  defence  industry,  the  Department  of  Defence,  and  academia  (Jacapino 
2010).  The  definitions  were  then  refined  with  reference  to  the  latest  literature  to  form  the  first 
published  version  that  was  used  to  support  the  interview  process  where  they  were  exposed, 
with  no  adverse  comment,  to  many  Defence  senior  executives.  The  definitions  were  exposed 
to  a  second  workshop  on  2  December  2010  and  the  final  definitions  that  appear  below 
incorporate  a  few  minor  changes  that  arose  from  the  salient  suggestions  captured  at  that 
event. 

The  definitions  given  below  are  intended  for  use  in  defence  capability  development, 
acquisition,  supplier,  and  operational  contexts. 

A.2  Definition  of  System  of  Systems 

In  the  Australian  Defence  context,  a  system  of  systems  (SoS)  is  a  large,  complex  system  that  exhibits 
the  following  two  essential  characteristics  that  distinguish  it  from  a  large,  monolithic  system: 

•  Operational  independence  of  component  systems  (Component  systems  are 
independent  and  useful  in  their  own  right.) 

•  Managerial  independence  of  the  component  systems  (Component  systems  are 
separately  acquired  and  integrated  but  maintain  a  continuing  operational  existence 
independent  of  the  SoS.  NB:  A  component  system  could  be  a  constituent  part  of  one  or  more 
SoS.) 

SoS  usually  exhibit  the  following  additional  properties: 


3  The  team  included  S  Cook,  R  Jeffery,  B  Madley,  T  Stevenson,  D  Williamson,  S  Soutberg,  A  Pitman  and 
A  Jacapino. 
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•  Evolutionary  development  (SoS  do  not  arrive  full  formed.  Development  is  evolutionary 
with  functions  and  purposes  added,  removed  or  modified  with  experience.) 

•  Emergent  behaviour  (A  SoS  performs  functions  and  carries  out  purposes  that  do  not 
wholly  reside  in  any  component  system.) 

•  Geographic  distribution  (The  geographic  extent  of  the  component  systems  is  often  large 
thus  SoS,  most  often,  can  readily  exchange  only  information  and  not  substantial  quantities 
of  mass  or  energy.) 

SoS  are  systems  at  a  level  of  hierarchy  beyond  what  we  normally  consider  in  a  single  Major  Capital 
Equipment  (MCE)  project.  In  this  context,  MCE  projects  give  rise  to  component  systems  whereas  SoS 
come  into  being  through  the  action  of  integrating  multiple  systems  that  were  not  necessarily  intended 
to  connect  together. 

A.3  Definition  of  High-End  Systems  Integration 

Systems  integration  can  have  several  meanings  as  outlined  in  this  missive  at  5.2.  It  can  be  a 
noun  that  labels  a  type  of  business,  an  adjective  that  qualifies  a  type  of  project,  or  a  noun  that 
relates  to  the  activity  of  doing  the  integration  phase  of  any  engineering  project  or  of  doing  a 
whole  systems  integration  project.  It  is  the  latter  definition  that  is  proffered  here. 

High-End  Systems  Integration  is  a  systems  engineering  approach  employed  to  realise  large,  complex, 
systems  (of  the  scale  and  complexity  tackled  by  ACAT  1  or  the  more  challenging  ACAT II  Major 
Capital  Equipment  projects)  that  places  the  major  emphasis  on  utilising  mostly  pre-existing 
components  and  focuses  the  engineering,  and  management  of  such  engineering,  on  the  inherent 
complexity  of  the  interfaces  between  those  components. 

A.4  Definition  of  System  of  Systems  Integration 

The  degree  of  difficulty  in  defining  this  term  comes  from  the  fact  that  there  are  at  least  four 
types  of  systems  of  systems  as  described  in  Section  5.1  and  that  many  SoS  are  integrated  upon 
deployment.  A  better  term  to  use  might  be  systems  of  systems  engineering  which  is  well 
understood,  enjoys  a  wide  usage,  and  for  which  there  are  a  number  of  guidebooks  (USN 
2006a  &  b).  It  is  also  important  to  understand  that  there  are  at  least  two  important 
perspectives  to  consider  on  SoS  integration.  The  first  is  the  enterprise  perspective  taken  by 
Defence,  or  similar  customer  and  operating  organisations  such  as  the  FAA,  that  is  concerned 
with  integrating  (military)  capabilities  across  all  fundamental  inputs  to  capability  as  described 
by  the  NCW  Integration  and  Implementation  Strategy  (Defence  2010).  NCWIIS  refers  to  this 
as  capability  integration  and  terms  such  as  capability  engineering  (INCOSE  UK  2010),  force-level 
systems  engineering,  enterprise  systems  engineering  are  often  applied  to  this  class  of  activity.  The 
second  perspective  is  the  supplier's  view  of  systems-of-systems  that,  while  aware  of  the 
broader  enterprise-level  integration  concerns,  is  focussed  on  the  technical  challenges  of 
integrating  disparate  systems  and  the  impacts  SoS  considerations  may  have  on  the  component 
systems  they  are  supplying  and  sustaining.  The  following  definition  attempts  to  cover  both 
perspectives: 

Systems  of  Systems  Integration  (SoSI)  is  a  systems  engineering  approach  to  planning,  analysing, 
organizing,  and  integrating  the  capabilities  of  a  mix  of  existing  and/or  new  systems  into  one  or  more 
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system-of-systems  capabilities  that  are  greater  than  the  sum  of  the  capabilities  of  the  constituent  parts. 
SoSI  emphasizes  the  process  of  discovering,  developing,  and  implementing  standards  that  promote 
interoperability  among  systems  developed  via  different  sponsorship  and  management  processes. 

The  framework  for  achieving  SoSI  within  the  Australian  Department  of  Defence  is  described  in  the 
NCW  Integration  and  Implementation  Strategy  that  outlines  an  enterprise  systems  engineering 
approach  to  achieving  capability  integration. 

It  is  recognised  that  SoSI  requires  the  integration  of  defence  enterprise  elements  and,  as  such, 
transcends  technical  concerns  and  embraces  all  Fundamental  Inputs  to  Capability. 

A.5  Background  Material 

A.5.1  Background  for  systems  of  systems  definition 


The  definition  provided  draws  on  Maier's  original  definition  of  systems  of  systems  (Maier 
1998): 

"Five  principal  characteristics  are  useful  in  distinguishing  very  large  and  complex  but 
monolithic  systems  from  true  systems-of-sy stems. 

1.  Operational  Independence  of  the  Elements:  If  the  system-of-systems  is  disassembled 
into  its  component  systems  the  component  systems  must  be  able  to  usefully  operate 
independently.  The  system-of-systems  is  composed  of  systems  which  are  independent 
and  useful  in  their  own  right. 

2.  Managerial  Independence  of  the  Elements:  The  component  systems  not  only  can 
operate  independently,  they  do  operate  independently.  The  component  systems  are 
separately  acquired  and  integrated  but  maintain  a  continuing  operational  existence 
independent  of  the  system-of-systems. 

3.  Evolutionary  Development:  The  system-of-systems  does  not  appear  fully  formed.  Its 
development  and  existence  is  evolutionary  with  functions  and  purposes  added,  removed, 
and  modified  with  experience. 

4.  Emergent  Behavior:  The  system  performs  functions  and  carries  out  purposes  that  do 
not  reside  in  any  component  system.  These  behaviors  are  emergent  properties  of  the 
entire  system-of-systems  and  cannot  be  localized  to  any  component  system.  The  principal 
purposes  of  the  systems-of-systems  are  fulfilled  by  these  behaviors. 

5.  Geographic  Distribution:  The  geographic  extent  of  the  component  systems  is  large. 

Large  is  a  nebulous  and  relative  concept  as  communication  capabilities  increase,  but  at  a 
minimum  it  means  that  the  components  can  readily  exchange  only  information  and  not 
substantial  quantities  of  mass  or  energy." 

Furthermore,  Maier  (1998)  seeks  to  identify  those  characteristics  that  distinguish  a  system  of 
systems  from  a  large,  complex  monolithic  system.  He  concludes  that  operational  and 
managerial  independence,  i.e.  the  first  two  characteristics,  are  sufficient  for  this  purpose  and 
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the  remaining  three,  while  usefully  descriptive,  are  not  able  to  assist  in  distinguishing  the  two 
types  of  system. 

It  is  important  to  be  aware  that  SoS  can  take  different  forms  and  as  such  different  approaches 
are  needed  to  guide  their  evolution  and  operation.  The  following  excerpt  summarises  this  for 
defence  applications  (OSD  2008): 

"In  DoD  and  elsewhere,  SoS  can  take  different  forms.  Based  on  a  recognized  taxonomy  of 
SoS,  there  are  four  types  of  SoS  which  are  found  in  the  DoD  today  [Maier,1998;  Dahmann, 

2008].  These  are: 

Virtual.  Virtual  SoS  lack  a  central  management  authority  and  a  centrally  agreed  upon 
purpose  for  the  system-of-sy stems.  Large-scale  behavior  emerges  — and  may  be 
desirable  — but  this  type  of  SoS  must  rely  upon  relatively  invisible  mechanisms  to 
maintain  it. 

Collaborative.  In  collaborative  SoS  the  component  systems  interact  more  or  less 
voluntarily  to  fulfill  agreed  upon  central  purposes.  The  Internet  is  a  collaborative  system. 

The  Internet  Engineering  Task  Force  works  out  standards  but  has  no  power  to  enforce 
them.  The  central  players  collectively  decide  how  to  provide  or  deny  service,  thereby 
providing  some  means  of  enforcing  and  maintaining  standards. 

Acknowledged.  Acknowledged  SoS  have  recognized  objectives,  a  designated  manager, 
and  resources  for  the  SoS;  however,  the  constituent  systems  retain  their  independent 
ownership,  objectives,  funding,  and  development  and  sustainment  approaches.  Changes 
in  the  systems  are  based  on  collaboration  between  the  SoS  and  the  system. 

Directed.  Directed  SoS  are  those  in  which  the  integrated  system-of-systems  is  built  and 
managed  to  fulfill  specific  purposes.  It  is  centrally  managed  during  long-term  operation 
to  continue  to  fulfill  those  purposes  as  well  as  any  new  ones  the  system  owners  might 
wish  to  address.  The  component  systems  maintain  an  ability  to  operate  independently, 
but  their  normal  operational  mode  is  subordinated  to  the  central  managed  purpose. 

This  characterization  offers  a  framework  for  understanding  SoS  in  the  DoD  today.  With 
the  advent  of  networks  and  increased  efforts  to  link  systems  for  information  sharing 
across  the  battle  space,  most  systems  are  part  of  virtual  SoS.  DoD  net-centric  policies  and 
strategies  have  attempted  to  provide  crosscutting  approaches  to  fostering  information 
sharing  in  the  absence  of  explicit  shared  objectives  or  management.  (See  section  1.5.2) 

As  users  and  systems  owners  understand  their  interdependencies,  there  are  increasing 
examples  of  collaborative  SoS  where  representatives  of  systems  choose  to  work  together 
for  their  mutual  benefit.  Communities  of  interest  (COI),  where  volunteers  come  together 
to  develop  ways  for  shared  interests  to  be  addressed  collaboratively  by  participants 
working  under  their  current  structures,  are  a  good  example. 

In  a  few  cases,  most  notably  Future  Combat  Systems,  a  common  objective  has  driven  the 
development  of  the  constituent  systems  from  the  outset.  Systems  in  this  category 
therefore  constitute  a  directed  SoS. 

In  the  DoD  today  we  see  a  growing  number  of  acknowledged  SoS.  Like  directed  SoS, 
acknowledged  SoS  have  recognized  authorities  and  resources  at  the  SoS  level.  However, 
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because  an  acknowledged  SoS  comprises  systems  that  maintain  independent  objectives, 
management,  and  resources,  along  with  independent  development  processes,  these  SoS 
are  largely  collaborative  in  practice.  For  systems  in  these  SoS,  in  particular,  their  normal 
operational  mode  is  not  subordinated  to  the  central  managed  purpose  —  a  distinct  feature 
of  a  directed  SoS.  Because  defense  acquisition  and  funding  are  still  largely  platform 
focused,  many  SoS  do  not  have  authority  over  the  systems,  and  they  typically  try  to 
address  SoS  objectives  by  leveraging  the  developments  of  the  systems,  which  are 
normally  more  long-standing  and  better  supported  than  the  SoS.  Consequently, 
acknowledged  SoS,  like  directed  SoS,  have  objectives,  management,  and  funding  without 
authority  over  the  constituent  systems.  Like  collaborative  SoS,  changes  in  systems  to  meet 
SoS  needs  are  based  on  agreement  and  collaboration,  not  top-down  authority  from  the 
SoS  manager." 

SoS  are  often  differentiated  from  large  monolithic  systems  by  comparison  illustrations  and 
tables,  see  Figure  A1  and  Table  A1  below. 


What’s  the  Difference? 


Systems 


System  of  Systems 


Figure  A1  Illustrative  difference  between  monolithic  systems  and  SoS  (Kaplan  2007) 
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Table  A1  Comparison  of  conventional  military  systems  and  systems  of  systems  (Cook  2001) 


System  Attribute 

Simple  Military  Systems 

Military  System-of-Systems 

Example 

Artillery  piece 

Combined  peace-keeping  force 

System  characteristics 

Mechanistic  but  human  directed. 

Socio-technical;  involves  people, 
organisations,  doctrine, 
equipment ... 

Number  of  system  elements 

Small 

Large 

Interaction  between  elements 

Few 

Many 

Attributes  of  Elements 

Predetermined 

Not  predetermined 

Interaction  between  elements 

Highly  organised 

Loosely  organised;  but 
interfaces  critical 

Behaviour 

Governed  by  well-defined  laws 

Probabilistic 

Evolution 

Does  not  evolve 

Evolves  over  time 

Nature  of  sub-systems 

Do  not  pursue  their  own  goals 

Are  purposeful  and  generate 
their  own  goals 

Interaction  with  environment 

Little 

Interacts  strongly 

System  Practice 

Requirements  elicitation 

Straightforward,  use  good 
systems  engineering  practice 

Problematical  because  of 
evolutionary  nature  and 
unanticipated  needs. 

Creation  methodology 

One  or  a  few  specific  acquisition 
actions 

Either  evolve  through  multiple 
acquisition  or  synthesised  and 
adapted  from  existing  inventory 
to  meet  an  unanticipated 
requirement 

Key  systems  philosophy 

Closed 

Open 

Key  design  philosophy 

Controlled  process 

Adaptability  and  flexibility 

UNCLASSIFIED 


53 


UNCLASSIFIED 

DSTO-TR-2743 

Enterprises  are  often  included  in  the  class  of  SoS  and  there  is  a  significant  literature  on 
Enterprise  Engineering  (BKCASE  2010)  that  describes  how  conventional  systems  engineering 
principals  can  be  extended  and  applied  to  enterprises.  Martin  (2010),  however,  makes  a 
distinction  between  enterprises  and  SoS  and  consequently  Enterprise  Systems  Engineering 
(ESE)  and  SoS  Systems  Engineering  (SoS  SE).  This  is  shown  pictorially  below  in  Figure  A2 
that  shows  that  SoS  can  span  enterprises  and  this  view  is  helpful  when  one  considers  SoS  SE 
from  a  supplier  perspective.  The  SoS  SE  perspective  has  a  greater  focus  on  technical  issues  and 
interoperability  than  it  does  certain  of  the  enterprise  extensions  to  conventional  SE  practice. 


Figure  A2.  Relationship  between  an  Enterprise  and  a  SoSs  (from  Martin,  2010) 

A.5.2  Background  for  High-End  Systems  Integration  Definition 

'High-end  systems  integration'  is  a  term  had  little  currency  before  the  advent  of  the  Priority 
Industry  Capabilities.  It  is  assumed  that  'high-end'  refers  to  the  complexity  and  scale  of  the 
systems  of  interest  in  a  Major  Capital  Equipment  project.  It  is  thus  reasonable  to  assume  that 
high-end  systems  integration  is  encountered  in  ACAT  1  systems  integration  projects  and 
perhaps  some  ACAT  2  projects.  See  Table  A2  for  a  definition  of  ACAT  levels. 
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Table  A2.  Definition  ofACAT  levels  from  the  Defence  Capability  Plan  (Defence  2009b) 


The  ACAT  framework  is  based  on  four  Acquisition  Categories  that  provide  a  graduated  scale  from  the 
most  demanding  and  complex  projects  to  those  that  are  less  so.  The  largest,  most  demanding  and 
complex  projects  are  categorised  as  ACAT  I  and  ACAT  II,  and  the  less  demanding  projects  are 
categorised  ACAT  III  and  ACAT  IV.  The  specific  description  of  each  level  is  as  follows: 

>  ACAT  I  projects  are  major  capital  equipment  acquisitions  that  are  normally  the  ADF’s  most 
strategically  significant.  They  are  characterised  by  extensive  project  and  schedule  management 
complexity  and  very  high  level  of  technical  difficulty,  operating,  support  and  commercial  arrangements. 

>  ACAT  II  projects  are  major  capital  equipment  acquisitions  that  are  strategically  significant  to  the  ADF. 
They  are  characterised  by  significant  project  and  schedule  management  complexity  and  high  levels  of 
technical  difficulty,  operating,  support  arrangements  and  commercial  arrangements. 

>  ACAT  III  projects  are  major  or  minor  capital  equipment  acquisitions  that  have  a  moderate  strategic 
significance  to  the  ADF.  They  are  characterised  by  the  application  of  traditional  project  and  schedule 
management  techniques  and  moderate  levels  of  technical  difficulty,  operating,  support  arrangements 
and  commercial  arrangements. 

>  ACAT  IV  projects  are  major  or  minor  capital  equipment  acquisitions  that  have  a  lower  level  of 
strategic  significance  to  the  ADF.  They  are  characterised  by  traditional  project  and  schedule 
management  requirements  and  lower  levels  of  technical  difficulty,  operating,  support  arrangements  and 
commercial  arrangements. 


A.5.3  Background  Information  on  the  Definition  of  SoS  Integration 
Some  helpful  definitions  follow: 

"System-of -Systems  Engineering  (SoSE)  is  defined  as:  The  process  of  planning, 
analyzing,  organizing,  and  integrating  the  capabilities  of  a  mix  of  existing  and  new 
systems  into  a  system-of-systems  capability  that  is  greater  than  the  sum  of  the  capabilities 
of  the  constituent  parts.  This  process  emphasizes  the  process  of  discovering,  developing, 
and  implementing  standards  that  promote  interoperability  among  systems  developed  via 
different  sponsorship,  management,  and  primary  acquisition  processes"  (USAF  2005). 

“...  we  shall  define  capability  engineering  as  a  way  of  conducting  capability  planning, 
architectural  design,  and  management  across  a  number  of  projects  that  draws  on  systems 
engineering  principles  for  its  philosophical  basis”  (Cook  &  Mun,  2006). 

“The  Naval  “Systems  of  Systems”  Systems  Engineering  Guidebook,  Volume  1,  Version 
1 .2  (originally  issued  as  the  Naval  Capabilities  Evolution  Process  Guidebook,  Volume  1) 
describes  a  comprehensive  process  for  applying  system-engineering  principles  that 
combine  the  capability  focus  of  JCIDS  with  the  evolutionary  acquisition  strategy  of  the 
Defense  Acquisition  System  to  evolve  to  a  networked  systems  environment.  This  Volume 
II  of  the  Guidebook  provides  an  initial  set  of  best  practices  that  can  be  applied  to 
implement  the  recommended  “systems  of  systems”  systems  engineering  process.  The 
intent  is  that  this  initial  set  will  be  significantly  augmented  by  recommended  best  practices 
for  capability-based  acquisition  and  systems  engineering  from  across  the  Naval  acquisition 
community.  In  this  regard,  your  best  practices  are  solicited  and  requested  to  be  submitted 
to  the  ASN  (RD&A)  Office  of  the  Chief  Engineer”  (Defence  2010). 
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“SoS  SE  is  defined  by  the  US  OSD  as  a  set  of  SE  activities  that  need  to  be  performed  at 
the  portfolio  level”  (Maier  1998). 

Enterprise  SE  seeks  to  lift  conventional  product-based  SE  to  the  enterprise  level  through 
broadening  the  scope  of  traditional  SE  process  and  work  products  and  by  additional  processes  that 
cover: 

•  Strategic  technical  planning 

•  Enterprise  architecture 

•  Capabilities-based  planning  analysis 

•  Technology  planning 

•  Enterprise  analysis 

The  INCOSE  UK  Capability  Working  Group  Perspectives  Analysis  Sub-Group  examined 
capability  engineering  from  the  perspective  of  eight  worldviews  (INCOSE  UK  2010)  encountered 
in  the  UK  on  what  comprises  capability  engineering.  This  approach  offers  the  advantage  of  not 
needing  to  come  to  a  compromise  definition  and  of  being  able  to  surface  a  broad  range  of 
perspectives. 

Figure  A3,  extracted  from  the  reference  illustrates  how  the  eight  worldviews  map  onto  five 
perspectives  and  Hitchins’  five  layers  of  systems  engineering  (Hitchins  2003). 


Figure  A3.  Weltanschauungen  (worldviews)  of  capability  engineering  as  related  to  the  Hitchins'  Five 
Layer  Model  of  systems  engineering 

As  one  would  expect,  different  stakeholders  hold  different  worldviews.  The  NCWIIS  takes  an 
enterprise  perspective  and  an  operational  perspective  as  these  perspectives  reflect  the  role  of 
members  of  the  Department  of  Defence.  Industry  on  the  other  hand  could  be  expected  to  take  a 
service  perspective,  where  they  see  capability  engineering  as  providing  a  service  to  a  customer,  and 
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a  product  perspective  where  they  need  to  consider  how  SoS  imperatives  map  onto  the  product 
systems  they  are  delivering  and  sustaining. 

This  figure  is  interesting  in  that  it  helps  explain  the  breadth  of  positions  of  what  constitutes 
capability  engineering.  It  is  also  useful  to  highlight  the  perspective  of  Plain  Old  Systems 
Engineering  (POSE)  subscribers  who  believe  that  capability  engineering  can  be  encompassed  by 
the  broad  church  of  systems  engineering  described  by  those  such  as  Hitchins  and  Hall  who  assert 
that  good  systems  engineering  concepts  can  inform  systems  engineering  activities  at  all  levels  of 
complexity.  Indeed  the  authors  cited  described  SoS  approaches  around  twenty  years  ago. 

In  formulating  the  definition  of  SoSI,  an  attempt  was  made  to  incorporate  both  the  supplier  view 
and  the  multiple  Defence  enterprise  views  as  articulated  by  the  NCWIIS. 
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Annex  B:  SoSE  Artefacts  for  Land  Force  Capability 

This  section  describes  each  of  the  SoS  SE  artefacts  discussed  e.  These  definitions  are  largely 
extracted  from  Dahmann  et  al  (2010)  and  supplemented  from  Dahmann  (2012)  but  have  been 
modified,  as  needed,  to  suit  the  Australian  Land  Force  Capability  Context.  The  descriptions 
also  include  a  discussion  of  how  each  artefact  is  used  in  the  SoS  SE  process. 

It  should  be  noted  that  constituent  systems  SPOs  implement  SE  for  their  systems  and  create 
artefacts  to  support  their  processes.  For  those  interested,  Dahmann  et  al  (2010)  provide  a 
useful  comparison  between  SoS  SE  artefacts  and  constituent  system  artefacts.  In  general,  SoS 
artefacts  address  comparable  issues  but  with  a  broader  focus  across  the  SoS  and  tend  to  take  a 
less  prescriptive  tone.  In  most  cases,  constituent  system  owners  and  the  engineering  team 
with  the  SPO  retain  responsibility  for  their  systems  even  when  they  are  part  of  a  SoS.  There  is 
no  intention  to  replicate  information  available  for  the  systems  in  SoS  artefacts  but  rather  to 
point  to  that  information  retained  by  systems  as  it  impacts  the  SoS. 

SoS  Capability  Objectives  are  a  statement  of  top  level  objectives  for  the  SoS.  They  describe 
the  capabilities  needed  by  the  user,  ideally  based  on  some  definitive  or  authoritative  materials 
(e.g..  White  Paper,  Defence  Capability  Planning  Guidance,  strategies  or  Australian  Capability 
Context  Scenarios).  They  are  used  by  the  SoS  Team  and  stakeholders  as  the  foundation  for  SoS 
requirements,  metrics,  etc.  The  capability  objectives  provide  a  basis  for  translating  operational 
needs  into  high  level  requirements,  assessing  performance  to  capability  objectives,  and 
developing  an  architecture  and  solution  options. 

The  SoS  CONOPS  describes  how  the  functionality  of  the  constituent  systems  in  the  SoS  will 
be  employed  in  an  operational  setting.  The  CONOPS  (CONcept  of  OPerationS)  is  developed 
by  the  prospective  operational  users  and  with  active  participation  from  the  SoS  Team  to 
describe  the  way  users  plan  to  operate  and  use  constituent  systems  to  achieve  the  objectives, 
as  influenced  by  the  various  environments  and  conditions  anticipated.  The  CONOPS  is 
developed  in  parallel  with  the  capability  objectives  and  as  the  capability  objectives  evolve,  the 
CONOPS  should  evolve  in  detail,  as  well.  The  SoS  Team  and  the  SPOs  use  the  CONOPS  to 
define  the  SoS  requirements  space,  to  identify  aspects  of  constituent  systems  which  could 
impact  the  SoS  design,  and  to  select  performance  metrics  and  test  environments. 

Systems  Information  is  accessed  and  organized  by  the  SoSE  Team  and  is  used  as  the  basis  for 
trades  as  the  SoS  evolves.  This  is  information  about  constituent  systems  that  impacts  SoS 
capability  objectives  and  includes  both  programmatic  and  technical  aspects  of  the  constituent 
systems  relevant  to  the  SoS.  The  information  comes  from  many  stakeholder  groups  including 
the  design  and  sustainment  staff  in  the  SPOs,  the  operators,  and  the  contractors  supporting 
the  system.  It  is  important  that  the  responsibility  for  producing  the  information  and  keeping 
up  to  date  be  allocated  to  the  SPOs  because  the  SoSE  Team  will  never  have  the  resources  to 
elicit  this  information  and  configuration  management  it.  It  also  must  fall  to  the  SPO  to  alert 
the  SoS  Team  whenever  there  are  changes  in  the  constituent  system  that  are  liable  to  impact 
the  SoS.  The  Agreements  (discussed  later)  need  to  include  this  service  from  the  SPOs  to  the 
SoS  Team.  This  information  assists  the  SoS  Team  to  understand  the  components  of  the  SoS, 
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including  technical,  organizational,  fiscal,  and  planning  perspectives.  The  information 
provides  the  basis  for  developing  and  evolving  the  SoS  architecture,  monitoring  and  assessing 
changes  to  both  the  SoS  and  individual  systems,  and  developing  SoS  capability  solution 
options. 

The  SoS  Requirements  Space  bounds  the  first-order  SoS  user  needs  (including  operational 
tasks  and  missions)  and  defines  the  functions  required  to  provide  the  capability  across  the 
range  of  user  environments  likely  to  be  encountered.  It  should  be  noted  that  this  artefact  is  a 
'requirements  space'  versus  a  set  of  'requirements',  because  in  a  SoS,  'requirements'  are  taken 
on  by  the  constituent  systems  to  meet  the  SoS.  The  requirements  space  includes  both  the  SoS 
Capability  Backlog  and  Problem  Reports.  It  is  developed  by  the  SoSE  Team  in  liaison  with 
constituent  system  SE  teams  and  with  strong  engagement  with  SoS  and  constituent  system 
operational  communities.  It  is  used  by  the  SoSE  Team  and  constituent  system  SE  teams  to: 
determine  information  needed  to  understand  systems  and  relationships,  compare 
performance  to  capability  objectives,  develop  an  SoS  architecture,  identify  areas  to  be 
addressed  in  an  increment(s),  identify  solution  options,  develop  a  plan  for  SoS  increment(s), 
and  develop  a  plan  to  test  and  evaluate  the  changes. 

SoS  Performance  Measures  and  Methods  provide  the  basis  for  assessing  overall  performance 
of  the  SoS  and  planning  for  'continuous  SoS  improvement'.  These  performance  measures  and 
methods  are  traceable  to  the  capability  objectives  established  for  the  SoS.  They  are  created  by 
the  SoSE  Team  and  the  constituent  system  SE  teams  together  with  the  test  and  evaluation 
(T&E)  community  to  assess  status  and  progress  in  meeting  SoS  capability  objectives  and  are 
used  to  structure  events  to  generate  the  data  needed. 

SoS  Performance  Data,  along  with  data  on  unanticipated  factors  observed  during 
performance  analysis,  are  gathered  from  different  environments  by  the  SoSE  Team  and  the 
T&E  teams  and  operators  to  assess  progress  toward  achieving  SoS  capability  objectives.  These 
data  are  used  by  SoS  management  and  SE  teams  to  assess  the  impact  of  changes  and  to 
identify  areas  needing  more  attention  (new  gaps/ requirements).  The  data  also  provide 
feedback  on  architecture  implementation  variability;  factors  impacting  capability;  and 
additional  capability  needs  based  on  operational  user  experience.  The  aggregate  feedback 
serves  as  a  basis  for  addressing  requirements  and  orchestrating  SoS  upgrades. 

SoS  SE  Planning  Elements  provide  the  structure  and  process  outline  for  SE  for  the  SoS  much 
as  a  System  Engineering  Plan  (SEP)  does  for  an  acquisition  program.  Key  elements  include  (1) 
battle  rhythm  or  pacing  of  SoS  upgrades,  (2)  organisational  structures  and  decision  processes, 
and  (3)  technical  reviews.  These  elements  are  developed  and  evolved  by  the  SoSE  Team  in 
conjunction  with  the  SE  teams  from  key  constituent  systems.  The  elements  provide  the  basic 
SE  rules  of  engagement  for  the  SoS  and  are  used  by  the  full  range  of  participants  in  the  SoS  to 
understand  the  overall  SoS  SE  process. 

SoS  Risks  and  Mitigations  are  addressed  throughout  the  process.  The  SoSE  Team  works  in 
collaboration  with  constituent  system  SE  teams  to  capture  the  potential  risks  associated  with 
SoS  capabilities  and  mitigations  for  them.  The  status  of  risks  and  their  mitigation  are  updated 
on  a  periodic  or  event-driven  basis  and  tracked  by  the  SoSE  Team,  constituent  system  SE 
teams,  and  SoS  stakeholders  to  understand  potential  risks,  issues,  and  obstacles  to  achieving 
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desired  capabilities  and  to  guide  selections  of  alternative  solutions.  SoS  risks  often  emanate 
from  areas  outside  the  SoS  where  changes  may  impact  SoS  objectives,  particularly  changes 
made  in  the  constituent  systems  to  meet  user  needs.  Monitoring  and  addressing  this  type  of 
risk  is  an  important  role  for  the  SoSE  Team. 

The  SoS  Master  Plan  is  an  integrated  plan  that  provides  a  top  level  view  across  multiple 
incremental  upgrades  to  implement  the  SoS  evolution  strategy;  it  is  the  SoS  analogue  to  a 
systems  acquisition  strategy.  This  plan  is  developed  and  evolved  by  the  SoSE  Team  in 
collaboration  with  constituent  system  SE  teams.  The  SoSE  Team,  constituent  system  SE  teams, 
and  SoS  stakeholders  use  it  to  understand  current  status  and  plans  of  the  SoS.  Since  this 
master  plan  looks  across  iterations  of  the  SoS,  it  provides  a  mechanism  for  supporting  trade¬ 
off  decisions  and  adjusting  priorities  over  time. 

Agreements  formalize  roles  and  responsibilities  of  SoS  participants  at  a  broad  level  (e.g. 
Charter)  as  well  as  specific  commitments  of  participants  in  a  development  increment.  Because 
SoS  cut  across  organizational  boundaries,  agreements  are  critical  to  SoSE  SE  success.  The  SoSE 
Team  and  constituent  system  SPOs  (and  contracting  officers  and  commercial  contractor 
representatives,  as  needed)  define  agreements  among  participants  regarding  organizational 
relationships,  roles,  and  responsibilities,  and  to  manage  interactions  of  participants  and  other 
stakeholders. 

SoS  Architecture  is  the  persistent  technical  framework  for  addressing  the  evolution  of  the  SoS 
to  meet  user  needs,  and  for  addressing  possible  changes  in  constituent  system  functionality, 
performance,  or  interfaces.  The  architecture  defines  the  way  the  systems  work  together  and 
addresses  the  implementation  of  individual  systems  only  when  the  functionality  is  key  to 
crosscutting  issues  of  the  SoS  (including  shared  data  specifications  or  data  models).  It  includes 
the  constituent  systems,  key  SoS  functions  supported  by  the  systems,  and  relationships  and 
dependencies  as  well  as  end-to-end  functionality,  data  flow,  and  communications  protocols. 
The  SoSE  Team  defines  the  desired  approach  to  organize  existing  and  newly  developed 
systems.  The  SoS  Team  and  the  constituent  system  SE  teams  use  the  architecture  products  as  a 
framework  for  developing  SoS  solutions.  It  provides  a  shared  representation  of  the  SoS 
technical  framework  used  to  inform  and  document  decisions  and  guide  evolution  of  the  SoS. 

SoS  Technical  Baselines  are  developed  for  each  increment  of  SoS  development.  These  SoS 
baselines  include  a  requirements  baseline,  an  allocated  baseline,  and  a  product  baseline  for  the 
SoS  and  reference  the  detailed  system  baselines  maintained  by  the  systems  themselves.  These 
are  used  to  understand  the  current  "as  is"  state  of  the  SoS  (product),  monitor  the  SoS 
enhancements  being  currently  developed  for  the  next  increment  (allocated),  and  plan  changes 
for  future  increments  (functional/ requirements). 

Technical  Plans  are  developed  for  each  development  increment  and  include  plans  for  SoS 
implementation,  integration,  and  test.  SoS  technical  plans  follow  the  principles  for  technical 
planning  for  systems,  paying  attention  to  defining  critical  event-driven  reviews  and  risks 
throughout  the  process.  The  SoSE  Team,  the  constituent  system  SPOs,  and  the  T&E 
community  use  these  plans  to  guide  activities  and  document  agreements  on  changes  to  be 
made  in  a  SoS  increment(s),  to  track  implementation  progress,  and  to  identify  changes/issues 
in  implementation. 
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Integrated  Master  Schedules  (IMS)  are  also  created  for  each  SoS  development  increment. 
They  include  the  key  points  in  the  technical  plans  which  need  to  be  addressed  in  orchestrating 
SoS  development.  The  IMS  focuses  on  key  SoSE  activities  and  integration  points  and  links  to 
the  detailed  development  schedules  maintained  by  the  systems  for  the  update.  In  the  IMS,  the 
SoSE  Team,  in  collaboration  with  constituent  system  SEs,  identify  key  activities  in  SoS  SE  as 
well  as  common  points  (synchronization  points,  critical  events)  across  elements  of  the  SoS  for 
an  increment(s).  The  SoSE  Team  and  constituent  system  SE  teams  use  the  IMS  to  monitor  key 
points  across  elements  of  the  SoS  within  the  increment(s). 

The  SoS  Test  and  Evaluation  Report  will  take  the  form  of  an  evaluation  of  the  degree  to 
which  the  SoS  achieves  the  capability  objectives  and  CONOPS  at  a  particular  point  in  time, 
either  at  a  particular  event  such  as  the  nominal  completion  of  an  iteration  or  at  a  calendar 
event  such  as  the  end  of  a  particular  year.  It  is  widely  recognised  that  it  is  impractical  to  test  a 
SoS  comprehensively  in  the  same  way  that  constituent  systems  are  tested  for  compliance  to  a 
substantial  requirement  specification  (Sage  &  Cuppan,  2001;  Dahmann  2012;  Kalawsky  2012). 
Thus  SoS  T&E  has  to  performed  on  the  basis  of  the  SoS  T&E  plan  and  the  emphasis  will  be  on 
synthesising  an  evaluation  of  the  SoS  based  on  constituent  systems  test  reports  and  models, 
SoS  modelling  and  simulation,  experimentation  reports,  operational  reports  and  limited  SoS 
testing. 
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